2024-03-29T10:58:43.101 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T10:58:43.101 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_amathuri@teuthology --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622677 --verbose --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622677/orig.config.yaml 2024-03-29T10:58:43.140 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T10:58:43.226 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622677 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622677' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi018.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIahkFEMlmnbD1gAQSJHalvyf71pZc2mxw+81x+ayQCdypgeL+kns4j8XKqBYveluo9sNsk0UJX9n5RTUJZ0Mnk= smithi070.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJrL8s29ynEsIThArj/rDyBt8CR2iFx2u+RvNeCjnlMFTVVtqNRzhhtNViVWYO5QIGCiXPV536ehJiwSbhIipA0= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T10:58:43.226 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T10:58:43.227 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T10:58:43.227 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T10:58:43.229 INFO:teuthology.task.internal:Checking packages... 2024-03-29T10:58:43.257 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T10:58:43.257 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T10:58:43.257 INFO:teuthology.packaging:ref: None 2024-03-29T10:58:43.257 INFO:teuthology.packaging:tag: None 2024-03-29T10:58:43.257 INFO:teuthology.packaging:branch: main 2024-03-29T10:58:43.257 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:58:43.257 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T10:58:43.574 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T10:58:43.576 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T10:58:43.582 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T10:58:43.582 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T10:58:43.591 INFO:teuthology.task.internal:Saving configuration 2024-03-29T10:58:43.605 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T10:58:43.610 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T10:58:43.641 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi018.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622677', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:53:50.297642', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIahkFEMlmnbD1gAQSJHalvyf71pZc2mxw+81x+ayQCdypgeL+kns4j8XKqBYveluo9sNsk0UJX9n5RTUJZ0Mnk='} 2024-03-29T10:58:43.661 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi070.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622677', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:53:50.299627', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJrL8s29ynEsIThArj/rDyBt8CR2iFx2u+RvNeCjnlMFTVVtqNRzhhtNViVWYO5QIGCiXPV536ehJiwSbhIipA0='} 2024-03-29T10:58:43.662 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T10:58:43.668 INFO:teuthology.task.internal:roles: ubuntu@smithi018.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T10:58:43.668 INFO:teuthology.task.internal:roles: ubuntu@smithi070.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T10:58:43.668 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T10:58:43.730 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1ba69890d0>, signals=[15]) 2024-03-29T10:58:43.730 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T10:58:43.736 INFO:teuthology.task.internal:Opening connections... 2024-03-29T10:58:43.736 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi018.front.sepia.ceph.com 2024-03-29T10:58:43.738 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi018.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:58:43.860 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi070.front.sepia.ceph.com 2024-03-29T10:58:43.861 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:58:43.983 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T10:58:43.989 DEBUG:teuthology.orchestra.run.smithi018:> uname -m 2024-03-29T10:58:44.007 INFO:teuthology.orchestra.run.smithi018.stdout:x86_64 2024-03-29T10:58:44.010 DEBUG:teuthology.orchestra.run.smithi018:> cat /etc/os-release 2024-03-29T10:58:44.065 INFO:teuthology.orchestra.run.smithi018.stdout:NAME="CentOS Stream" 2024-03-29T10:58:44.065 INFO:teuthology.orchestra.run.smithi018.stdout:VERSION="9" 2024-03-29T10:58:44.065 INFO:teuthology.orchestra.run.smithi018.stdout:ID="centos" 2024-03-29T10:58:44.065 INFO:teuthology.orchestra.run.smithi018.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:58:44.065 INFO:teuthology.orchestra.run.smithi018.stdout:VERSION_ID="9" 2024-03-29T10:58:44.065 INFO:teuthology.orchestra.run.smithi018.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:58:44.065 INFO:teuthology.orchestra.run.smithi018.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:58:44.066 INFO:teuthology.orchestra.run.smithi018.stdout:ANSI_COLOR="0;31" 2024-03-29T10:58:44.066 INFO:teuthology.orchestra.run.smithi018.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:58:44.066 INFO:teuthology.orchestra.run.smithi018.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:58:44.066 INFO:teuthology.orchestra.run.smithi018.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:58:44.066 INFO:teuthology.orchestra.run.smithi018.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:58:44.066 INFO:teuthology.orchestra.run.smithi018.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:58:44.066 INFO:teuthology.orchestra.run.smithi018.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:58:44.067 INFO:teuthology.lock.ops:Updating smithi018.front.sepia.ceph.com on lock server 2024-03-29T10:58:44.092 DEBUG:teuthology.orchestra.run.smithi070:> uname -m 2024-03-29T10:58:44.109 INFO:teuthology.orchestra.run.smithi070.stdout:x86_64 2024-03-29T10:58:44.109 DEBUG:teuthology.orchestra.run.smithi070:> cat /etc/os-release 2024-03-29T10:58:44.165 INFO:teuthology.orchestra.run.smithi070.stdout:NAME="CentOS Stream" 2024-03-29T10:58:44.166 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION="9" 2024-03-29T10:58:44.166 INFO:teuthology.orchestra.run.smithi070.stdout:ID="centos" 2024-03-29T10:58:44.166 INFO:teuthology.orchestra.run.smithi070.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:58:44.166 INFO:teuthology.orchestra.run.smithi070.stdout:VERSION_ID="9" 2024-03-29T10:58:44.166 INFO:teuthology.orchestra.run.smithi070.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:58:44.166 INFO:teuthology.orchestra.run.smithi070.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:58:44.166 INFO:teuthology.orchestra.run.smithi070.stdout:ANSI_COLOR="0;31" 2024-03-29T10:58:44.166 INFO:teuthology.orchestra.run.smithi070.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:58:44.166 INFO:teuthology.orchestra.run.smithi070.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:58:44.166 INFO:teuthology.orchestra.run.smithi070.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:58:44.166 INFO:teuthology.orchestra.run.smithi070.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:58:44.167 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:58:44.167 INFO:teuthology.orchestra.run.smithi070.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:58:44.167 INFO:teuthology.lock.ops:Updating smithi070.front.sepia.ceph.com on lock server 2024-03-29T10:58:44.192 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T10:58:44.201 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T10:58:44.207 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T10:58:44.207 DEBUG:teuthology.orchestra.run.smithi018:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:58:44.209 DEBUG:teuthology.orchestra.run.smithi070:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:58:44.224 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T10:58:44.232 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T10:58:44.232 DEBUG:teuthology.orchestra.run.smithi018:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:58:44.266 DEBUG:teuthology.orchestra.run.smithi070:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:58:44.290 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T10:58:44.344 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T10:58:44.360 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T10:58:44.360 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T10:58:44.360 DEBUG:teuthology.orchestra.run.smithi018:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:58:44.361 DEBUG:teuthology.orchestra.run.smithi070:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:58:44.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:58:44.376 DEBUG:teuthology.orchestra.run.smithi070:> uname -r 2024-03-29T10:58:44.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:58:44.377 DEBUG:teuthology.orchestra.run.smithi018:> uname -r 2024-03-29T10:58:44.433 INFO:teuthology.orchestra.run.smithi070.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:58:44.433 INFO:teuthology.task.kernel:Running kernel on smithi070: 5.14.0-361.el9.x86_64 2024-03-29T10:58:44.433 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum install -y kernel 2024-03-29T10:58:44.434 INFO:teuthology.orchestra.run.smithi018.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:58:44.434 INFO:teuthology.task.kernel:Running kernel on smithi018: 5.14.0-361.el9.x86_64 2024-03-29T10:58:44.434 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum install -y kernel 2024-03-29T10:58:46.445 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-29T10:58:46.532 INFO:teuthology.orchestra.run.smithi018.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-29T10:58:47.999 INFO:teuthology.orchestra.run.smithi018.stdout:CentOS-9 - BaseOS 6.4 MB/s | 2.0 MB 00:00 2024-03-29T10:58:48.070 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-29T10:58:48.819 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - BaseOS 20 MB/s | 8.0 MB 00:00 2024-03-29T10:58:49.007 INFO:teuthology.orchestra.run.smithi018.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-29T10:58:50.866 INFO:teuthology.orchestra.run.smithi018.stdout:CentOS Stream 9 - AppStream 27 MB/s | 19 MB 00:00 2024-03-29T10:58:56.075 INFO:teuthology.orchestra.run.smithi018.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 6.1 MB 00:01 2024-03-29T10:58:57.500 INFO:teuthology.orchestra.run.smithi018.stdout:CentOS Stream 9 - Extras packages 76 kB/s | 16 kB 00:00 2024-03-29T10:58:59.446 INFO:teuthology.orchestra.run.smithi018.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-29T10:59:00.159 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - AppStream 1.8 MB/s | 19 MB 00:10 2024-03-29T10:59:04.602 INFO:teuthology.orchestra.run.smithi018.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2024-03-29T10:59:04.626 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - CRB 6.4 MB/s | 6.1 MB 00:00 2024-03-29T10:59:06.984 INFO:teuthology.orchestra.run.smithi018.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:59:06.984 INFO:teuthology.orchestra.run.smithi018.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:59:07.078 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-29T10:59:07.090 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-29T10:59:07.090 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repository Size 2024-03-29T10:59:07.090 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-29T10:59:07.090 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-29T10:59:07.090 INFO:teuthology.orchestra.run.smithi018.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:59:07.090 INFO:teuthology.orchestra.run.smithi018.stdout:Installing dependencies: 2024-03-29T10:59:07.090 INFO:teuthology.orchestra.run.smithi018.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:59:07.090 INFO:teuthology.orchestra.run.smithi018.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:59:07.091 INFO:teuthology.orchestra.run.smithi018.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:59:07.091 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T10:59:07.091 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-29T10:59:07.091 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-29T10:59:07.091 INFO:teuthology.orchestra.run.smithi018.stdout:Install 4 Packages 2024-03-29T10:59:07.091 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T10:59:07.091 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 76 M 2024-03-29T10:59:07.091 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 123 M 2024-03-29T10:59:07.091 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-29T10:59:07.177 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - Extras packages 11 kB/s | 16 kB 00:01 2024-03-29T10:59:07.245 INFO:teuthology.orchestra.run.smithi018.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-29T10:59:07.878 INFO:teuthology.orchestra.run.smithi018.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 19 MB/s | 15 MB 00:00 2024-03-29T10:59:08.395 INFO:teuthology.orchestra.run.smithi070.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-03-29T10:59:08.620 INFO:teuthology.orchestra.run.smithi018.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 22 MB/s | 34 MB 00:01 2024-03-29T10:59:08.895 INFO:teuthology.orchestra.run.smithi018.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 17 MB/s | 28 MB 00:01 2024-03-29T10:59:08.898 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:59:08.899 INFO:teuthology.orchestra.run.smithi018.stdout:Total 42 MB/s | 76 MB 00:01 2024-03-29T10:59:08.899 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-29T10:59:09.331 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-29T10:59:09.331 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-29T10:59:10.184 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-29T10:59:10.185 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-29T10:59:12.609 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-29T10:59:13.234 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:59:13.311 INFO:teuthology.orchestra.run.smithi070.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-03-29T10:59:13.329 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:59:15.567 INFO:teuthology.orchestra.run.smithi070.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:59:15.567 INFO:teuthology.orchestra.run.smithi070.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:59:15.657 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-03-29T10:59:15.662 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-03-29T10:59:15.663 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-03-29T10:59:15.663 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-03-29T10:59:15.663 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-03-29T10:59:15.663 INFO:teuthology.orchestra.run.smithi070.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:59:15.663 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-03-29T10:59:15.663 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:59:15.663 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:59:15.663 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:59:15.663 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T10:59:15.663 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-03-29T10:59:15.663 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-03-29T10:59:15.663 INFO:teuthology.orchestra.run.smithi070.stdout:Install 4 Packages 2024-03-29T10:59:15.663 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T10:59:15.664 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 76 M 2024-03-29T10:59:15.664 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 123 M 2024-03-29T10:59:15.664 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-03-29T10:59:15.812 INFO:teuthology.orchestra.run.smithi070.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-03-29T10:59:16.238 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:59:16.304 INFO:teuthology.orchestra.run.smithi070.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 24 MB/s | 15 MB 00:00 2024-03-29T10:59:16.374 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:59:16.871 INFO:teuthology.orchestra.run.smithi070.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 28 MB/s | 34 MB 00:01 2024-03-29T10:59:17.162 INFO:teuthology.orchestra.run.smithi070.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 20 MB/s | 28 MB 00:01 2024-03-29T10:59:17.165 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:59:17.166 INFO:teuthology.orchestra.run.smithi070.stdout:Total 51 MB/s | 76 MB 00:01 2024-03-29T10:59:17.166 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-03-29T10:59:17.552 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-03-29T10:59:17.552 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-03-29T10:59:18.383 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-03-29T10:59:18.384 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-03-29T10:59:20.671 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-03-29T10:59:21.270 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:59:21.363 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:59:21.445 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:59:21.482 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:59:24.131 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:59:24.272 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:59:26.585 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:59:28.996 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:59:29.031 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:59:33.791 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:59:57.396 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:59:57.396 INFO:teuthology.orchestra.run.smithi018.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T10:59:57.396 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T10:59:57.441 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:59:57.817 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:59:57.817 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:59:57.817 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:59:57.817 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:59:58.235 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:59:58.235 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T10:59:58.235 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-29T10:59:58.235 INFO:teuthology.orchestra.run.smithi018.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T10:59:58.236 INFO:teuthology.orchestra.run.smithi018.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T10:59:58.236 INFO:teuthology.orchestra.run.smithi018.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T10:59:58.236 INFO:teuthology.orchestra.run.smithi018.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T10:59:58.236 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T10:59:58.236 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-29T10:59:58.562 DEBUG:teuthology.orchestra.run.smithi018:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:59:58.667 INFO:teuthology.orchestra.run.smithi018.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:59:58.668 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:59:58.668 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T10:59:58.668 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T10:59:58.668 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T10:59:58.668 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T10:59:58.668 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi018.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T10:59:58.669 DEBUG:teuthology.orchestra.run.smithi018:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:59:58.751 INFO:teuthology.orchestra.run.smithi018.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:59:58.751 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:59:58.752 DEBUG:teuthology.orchestra.run.smithi018:> sudo rpm -qi grub2-tools 2024-03-29T10:59:58.790 INFO:teuthology.orchestra.run.smithi018.stdout:Name : grub2-tools 2024-03-29T10:59:58.790 INFO:teuthology.orchestra.run.smithi018.stdout:Epoch : 1 2024-03-29T10:59:58.790 INFO:teuthology.orchestra.run.smithi018.stdout:Version : 2.06 2024-03-29T10:59:58.790 INFO:teuthology.orchestra.run.smithi018.stdout:Release : 68.el9 2024-03-29T10:59:58.790 INFO:teuthology.orchestra.run.smithi018.stdout:Architecture: x86_64 2024-03-29T10:59:58.790 INFO:teuthology.orchestra.run.smithi018.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T10:59:58.790 INFO:teuthology.orchestra.run.smithi018.stdout:Group : Unspecified 2024-03-29T10:59:58.790 INFO:teuthology.orchestra.run.smithi018.stdout:Size : 8264254 2024-03-29T10:59:58.790 INFO:teuthology.orchestra.run.smithi018.stdout:License : GPLv3+ 2024-03-29T10:59:58.791 INFO:teuthology.orchestra.run.smithi018.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T10:59:58.791 INFO:teuthology.orchestra.run.smithi018.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T10:59:58.791 INFO:teuthology.orchestra.run.smithi018.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T10:59:58.791 INFO:teuthology.orchestra.run.smithi018.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T10:59:58.791 INFO:teuthology.orchestra.run.smithi018.stdout:Packager : builder@centos.org 2024-03-29T10:59:58.791 INFO:teuthology.orchestra.run.smithi018.stdout:Vendor : CentOS 2024-03-29T10:59:58.791 INFO:teuthology.orchestra.run.smithi018.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T10:59:58.791 INFO:teuthology.orchestra.run.smithi018.stdout:Summary : Support tools for GRUB. 2024-03-29T10:59:58.791 INFO:teuthology.orchestra.run.smithi018.stdout:Description : 2024-03-29T10:59:58.791 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T10:59:58.791 INFO:teuthology.orchestra.run.smithi018.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T10:59:58.791 INFO:teuthology.orchestra.run.smithi018.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T10:59:58.791 INFO:teuthology.orchestra.run.smithi018.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T10:59:58.792 INFO:teuthology.orchestra.run.smithi018.stdout:hardware devices. 2024-03-29T10:59:58.792 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T10:59:58.792 INFO:teuthology.orchestra.run.smithi018.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T10:59:58.792 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T10:59:58.792 INFO:teuthology.task.kernel:Updating grub on smithi018 to boot 5.14.0-434.el9.x86_64 2024-03-29T10:59:58.793 DEBUG:teuthology.orchestra.run.smithi018:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T10:59:59.412 INFO:teuthology.orchestra.run.smithi018.stderr:Generating grub configuration file ... 2024-03-29T11:00:00.854 INFO:teuthology.orchestra.run.smithi018.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:00:00.873 INFO:teuthology.orchestra.run.smithi018.stderr:done 2024-03-29T11:00:00.875 DEBUG:teuthology.orchestra.run.smithi018:> mktemp 2024-03-29T11:00:00.891 INFO:teuthology.orchestra.run.smithi018.stdout:/tmp/tmp.XswSCH3zQi 2024-03-29T11:00:00.891 DEBUG:teuthology.orchestra.run.smithi018:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.XswSCH3zQi 2024-03-29T11:00:00.963 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod 0666 /tmp/tmp.XswSCH3zQi 2024-03-29T11:00:01.099 DEBUG:teuthology.orchestra.remote:smithi018:/tmp/tmp.XswSCH3zQi is 6KB 2024-03-29T11:00:01.147 DEBUG:teuthology.orchestra.run.smithi018:> rm -fr /tmp/tmp.XswSCH3zQi 2024-03-29T11:00:01.162 DEBUG:teuthology.orchestra.run.smithi018:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:00:01.230 INFO:teuthology.orchestra.run.smithi018.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:00:01.230 INFO:teuthology.orchestra.run.smithi018.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:00:01.231 INFO:teuthology.orchestra.run.smithi018.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:00:01.231 INFO:teuthology.orchestra.run.smithi018.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:00:01.232 DEBUG:teuthology.orchestra.run.smithi018:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:00:01.388 DEBUG:teuthology.orchestra.run.smithi018:> sudo shutdown -r now 2024-03-29T11:00:01.468 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2024-03-29T11:00:03.044 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:00:03.044 INFO:teuthology.orchestra.run.smithi070.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:00:03.044 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:00:03.087 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:00:03.420 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:00:03.420 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:00:03.420 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:00:03.420 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:00:03.805 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:00:03.805 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:00:03.806 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-03-29T11:00:03.806 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:00:03.806 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:00:03.806 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:00:03.806 INFO:teuthology.orchestra.run.smithi070.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:00:03.806 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:00:03.806 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-03-29T11:00:04.165 DEBUG:teuthology.orchestra.run.smithi070:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:00:04.264 INFO:teuthology.orchestra.run.smithi070.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:00:04.264 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:00:04.264 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:00:04.264 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:00:04.264 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:00:04.264 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T11:00:04.264 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi070.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:00:04.265 DEBUG:teuthology.orchestra.run.smithi070:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:00:04.344 INFO:teuthology.orchestra.run.smithi070.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:00:04.344 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:00:04.344 DEBUG:teuthology.orchestra.run.smithi070:> sudo rpm -qi grub2-tools 2024-03-29T11:00:04.381 INFO:teuthology.orchestra.run.smithi070.stdout:Name : grub2-tools 2024-03-29T11:00:04.382 INFO:teuthology.orchestra.run.smithi070.stdout:Epoch : 1 2024-03-29T11:00:04.382 INFO:teuthology.orchestra.run.smithi070.stdout:Version : 2.06 2024-03-29T11:00:04.382 INFO:teuthology.orchestra.run.smithi070.stdout:Release : 68.el9 2024-03-29T11:00:04.382 INFO:teuthology.orchestra.run.smithi070.stdout:Architecture: x86_64 2024-03-29T11:00:04.382 INFO:teuthology.orchestra.run.smithi070.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:00:04.382 INFO:teuthology.orchestra.run.smithi070.stdout:Group : Unspecified 2024-03-29T11:00:04.382 INFO:teuthology.orchestra.run.smithi070.stdout:Size : 8264254 2024-03-29T11:00:04.382 INFO:teuthology.orchestra.run.smithi070.stdout:License : GPLv3+ 2024-03-29T11:00:04.382 INFO:teuthology.orchestra.run.smithi070.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:00:04.383 INFO:teuthology.orchestra.run.smithi070.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:00:04.383 INFO:teuthology.orchestra.run.smithi070.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:00:04.383 INFO:teuthology.orchestra.run.smithi070.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:00:04.383 INFO:teuthology.orchestra.run.smithi070.stdout:Packager : builder@centos.org 2024-03-29T11:00:04.383 INFO:teuthology.orchestra.run.smithi070.stdout:Vendor : CentOS 2024-03-29T11:00:04.383 INFO:teuthology.orchestra.run.smithi070.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:00:04.383 INFO:teuthology.orchestra.run.smithi070.stdout:Summary : Support tools for GRUB. 2024-03-29T11:00:04.383 INFO:teuthology.orchestra.run.smithi070.stdout:Description : 2024-03-29T11:00:04.383 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:00:04.383 INFO:teuthology.orchestra.run.smithi070.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:00:04.383 INFO:teuthology.orchestra.run.smithi070.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:00:04.384 INFO:teuthology.orchestra.run.smithi070.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:00:04.384 INFO:teuthology.orchestra.run.smithi070.stdout:hardware devices. 2024-03-29T11:00:04.384 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:00:04.384 INFO:teuthology.orchestra.run.smithi070.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:00:04.385 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:00:04.385 INFO:teuthology.task.kernel:Updating grub on smithi070 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:00:04.385 DEBUG:teuthology.orchestra.run.smithi070:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:00:05.025 INFO:teuthology.orchestra.run.smithi070.stderr:Generating grub configuration file ... 2024-03-29T11:00:06.486 INFO:teuthology.orchestra.run.smithi070.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:00:06.516 INFO:teuthology.orchestra.run.smithi070.stderr:done 2024-03-29T11:00:06.519 DEBUG:teuthology.orchestra.run.smithi070:> mktemp 2024-03-29T11:00:06.534 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/tmp.0ndZ8sdjAw 2024-03-29T11:00:06.534 DEBUG:teuthology.orchestra.run.smithi070:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.0ndZ8sdjAw 2024-03-29T11:00:06.603 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 0666 /tmp/tmp.0ndZ8sdjAw 2024-03-29T11:00:06.739 DEBUG:teuthology.orchestra.remote:smithi070:/tmp/tmp.0ndZ8sdjAw is 6KB 2024-03-29T11:00:06.787 DEBUG:teuthology.orchestra.run.smithi070:> rm -fr /tmp/tmp.0ndZ8sdjAw 2024-03-29T11:00:06.802 DEBUG:teuthology.orchestra.run.smithi070:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:00:06.874 INFO:teuthology.orchestra.run.smithi070.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:00:06.874 INFO:teuthology.orchestra.run.smithi070.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:00:06.874 INFO:teuthology.orchestra.run.smithi070.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:00:06.874 INFO:teuthology.orchestra.run.smithi070.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:00:06.876 DEBUG:teuthology.orchestra.run.smithi070:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:00:07.043 DEBUG:teuthology.orchestra.run.smithi070:> sudo shutdown -r now 2024-03-29T11:00:31.415 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T11:00:31.416 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2024-03-29T11:00:31.417 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi018.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:00:37.052 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T11:00:37.052 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-03-29T11:00:37.053 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:00:49.823 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.18 2024-03-29T11:00:55.454 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.70 2024-03-29T11:00:58.827 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2024-03-29T11:00:58.828 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi018.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:01:04.459 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-03-29T11:01:04.460 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:01:07.518 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.70 2024-03-29T11:01:19.530 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-03-29T11:01:19.531 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:01:30.611 DEBUG:teuthology.orchestra.run.smithi018:> true 2024-03-29T11:01:31.050 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi018.front.sepia.ceph.com' 2024-03-29T11:01:31.050 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:01:31.050 DEBUG:teuthology.orchestra.run.smithi018:> uname -r 2024-03-29T11:01:31.065 INFO:teuthology.orchestra.run.smithi018.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:01:31.065 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:01:31.065 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:01:31.066 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:01:32.066 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T11:01:32.066 DEBUG:teuthology.orchestra.run.smithi018:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:01:32.118 INFO:teuthology.orchestra.run.smithi018.stdout:ttyS1 2024-03-29T11:01:32.157 DEBUG:teuthology.parallel:result is None 2024-03-29T11:01:51.258 DEBUG:teuthology.orchestra.run.smithi070:> true 2024-03-29T11:01:51.949 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-03-29T11:01:51.949 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:01:51.949 DEBUG:teuthology.orchestra.run.smithi070:> uname -r 2024-03-29T11:01:51.963 INFO:teuthology.orchestra.run.smithi070.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:01:51.963 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:01:51.963 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:01:51.963 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:01:52.964 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T11:01:52.964 DEBUG:teuthology.orchestra.run.smithi070:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:01:53.051 INFO:teuthology.orchestra.run.smithi070.stdout:ttyS1 2024-03-29T11:01:53.089 DEBUG:teuthology.parallel:result is None 2024-03-29T11:01:53.090 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T11:01:53.102 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T11:01:53.102 DEBUG:teuthology.orchestra.run.smithi018:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:01:53.106 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:01:53.123 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T11:01:53.130 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T11:01:53.140 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T11:01:53.140 DEBUG:teuthology.orchestra.run.smithi018:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:01:53.166 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:01:53.198 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T11:01:53.206 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T11:01:53.207 DEBUG:teuthology.orchestra.run.smithi018:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:01:53.232 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:01:53.262 INFO:teuthology.orchestra.run.smithi018.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:01:53.272 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:01:53.276 INFO:teuthology.orchestra.run.smithi018.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:01:53.286 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:01:53.288 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T11:01:53.296 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T11:01:53.296 DEBUG:teuthology.orchestra.run.smithi018:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:01:53.321 DEBUG:teuthology.orchestra.run.smithi070:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:01:53.358 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T11:01:53.369 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T11:01:53.369 DEBUG:teuthology.orchestra.run.smithi018:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:01:53.390 DEBUG:teuthology.orchestra.run.smithi070:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:01:53.412 DEBUG:teuthology.orchestra.run.smithi018:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:01:53.488 DEBUG:teuthology.orchestra.run.smithi018:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:01:53.539 DEBUG:teuthology.orchestra.run.smithi018:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:01:53.620 DEBUG:teuthology.orchestra.run.smithi018:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:01:53.664 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-29T11:01:53.665 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:01:53.732 DEBUG:teuthology.orchestra.run.smithi070:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:01:53.777 DEBUG:teuthology.orchestra.run.smithi070:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:01:53.829 DEBUG:teuthology.orchestra.run.smithi070:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:01:53.912 DEBUG:teuthology.orchestra.run.smithi070:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:01:53.956 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-03-29T11:01:53.956 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:01:54.023 DEBUG:teuthology.orchestra.run.smithi018:> sudo service rsyslog restart 2024-03-29T11:01:54.026 DEBUG:teuthology.orchestra.run.smithi070:> sudo service rsyslog restart 2024-03-29T11:01:54.073 INFO:teuthology.orchestra.run.smithi018.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:01:54.112 INFO:teuthology.orchestra.run.smithi070.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:01:54.492 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T11:01:54.499 INFO:teuthology.task.internal:Starting timer... 2024-03-29T11:01:54.499 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T11:01:54.510 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T11:01:54.520 DEBUG:teuthology.orchestra.run.smithi018:> sudo service auditd rotate 2024-03-29T11:01:54.585 INFO:teuthology.orchestra.run.smithi018.stdout:Rotating logs: 2024-03-29T11:01:54.587 DEBUG:teuthology.orchestra.run.smithi070:> sudo service auditd rotate 2024-03-29T11:01:54.649 INFO:teuthology.orchestra.run.smithi070.stdout:Rotating logs: 2024-03-29T11:01:54.651 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T11:01:54.651 DEBUG:teuthology.orchestra.run.smithi018:> /usr/sbin/getenforce 2024-03-29T11:01:54.671 INFO:teuthology.orchestra.run.smithi018.stdout:Permissive 2024-03-29T11:01:54.672 DEBUG:teuthology.orchestra.run.smithi070:> /usr/sbin/getenforce 2024-03-29T11:01:54.704 INFO:teuthology.orchestra.run.smithi070.stdout:Permissive 2024-03-29T11:01:54.704 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi018.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi070.front.sepia.ceph.com': 'permissive'} 2024-03-29T11:01:54.705 DEBUG:teuthology.orchestra.run.smithi018:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:01:54.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:01:54.741 DEBUG:teuthology.orchestra.run.smithi070:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:01:54.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:01:54.771 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T11:01:54.771 DEBUG:teuthology.orchestra.run.smithi018:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:01:54.823 DEBUG:teuthology.orchestra.run.smithi070:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:01:54.856 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T11:01:54.866 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T11:01:54.881 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T11:01:54.882 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi018.front.sepia.ceph.com,smithi070.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T11:06:10.915 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi018.front.sepia.ceph.com'), Remote(name='ubuntu@smithi070.front.sepia.ceph.com')] 2024-03-29T11:06:10.917 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi018.front.sepia.ceph.com' 2024-03-29T11:06:10.918 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi018.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:06:11.007 DEBUG:teuthology.orchestra.run.smithi018:> true 2024-03-29T11:06:11.103 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi018.front.sepia.ceph.com' 2024-03-29T11:06:11.104 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-03-29T11:06:11.104 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi070.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:06:11.193 DEBUG:teuthology.orchestra.run.smithi070:> true 2024-03-29T11:06:11.278 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi070.front.sepia.ceph.com' 2024-03-29T11:06:11.278 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T11:06:11.289 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T11:06:11.290 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:06:11.290 DEBUG:teuthology.orchestra.run.smithi018:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:06:11.293 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:06:11.293 DEBUG:teuthology.orchestra.run.smithi070:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:06:11.324 INFO:teuthology.orchestra.run.smithi018.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:06:11.341 INFO:teuthology.orchestra.run.smithi018.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:06:11.350 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:06:11.368 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:06:11.372 INFO:teuthology.orchestra.run.smithi018.stderr:sudo: ntpd: command not found 2024-03-29T11:06:11.386 INFO:teuthology.orchestra.run.smithi018.stdout:506 Cannot talk to daemon 2024-03-29T11:06:11.397 INFO:teuthology.orchestra.run.smithi070.stderr:sudo: ntpd: command not found 2024-03-29T11:06:11.403 INFO:teuthology.orchestra.run.smithi018.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:06:11.410 INFO:teuthology.orchestra.run.smithi070.stdout:506 Cannot talk to daemon 2024-03-29T11:06:11.419 INFO:teuthology.orchestra.run.smithi018.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:06:11.427 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:06:11.443 INFO:teuthology.orchestra.run.smithi070.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:06:11.476 INFO:teuthology.orchestra.run.smithi018.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:06:11.479 INFO:teuthology.orchestra.run.smithi018.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:06:11.479 INFO:teuthology.orchestra.run.smithi018.stdout:=============================================================================== 2024-03-29T11:06:11.479 INFO:teuthology.orchestra.run.smithi018.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:06:11.479 INFO:teuthology.orchestra.run.smithi018.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:06:11.479 INFO:teuthology.orchestra.run.smithi018.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:06:11.479 INFO:teuthology.orchestra.run.smithi018.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:06:11.497 INFO:teuthology.orchestra.run.smithi070.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:06:11.501 INFO:teuthology.orchestra.run.smithi070.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:06:11.501 INFO:teuthology.orchestra.run.smithi070.stdout:=============================================================================== 2024-03-29T11:06:11.501 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:06:11.502 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:06:11.502 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:06:11.502 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:06:11.502 INFO:teuthology.run_tasks:Running task install... 2024-03-29T11:06:11.512 DEBUG:teuthology.task.install:project ceph 2024-03-29T11:06:11.512 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T11:06:11.512 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T11:06:11.512 INFO:teuthology.task.install:Using flavor: default 2024-03-29T11:06:11.518 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T11:06:11.518 INFO:teuthology.task.install:extra packages: [] 2024-03-29T11:06:11.519 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:06:11.519 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:06:11.521 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:06:11.521 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:06:11.778 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:06:11.778 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:06:11.782 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:06:11.782 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:06:11.887 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:06:11.887 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-29T11:06:11.887 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:06:11.894 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:06:11.895 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-03-29T11:06:11.895 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:06:11.918 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:06:11.918 DEBUG:teuthology.orchestra.run.smithi018:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:06:11.922 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:06:11.922 DEBUG:teuthology.orchestra.run.smithi070:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:06:11.987 DEBUG:teuthology.orchestra.run.smithi018:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:06:11.990 DEBUG:teuthology.orchestra.run.smithi070:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:06:12.068 DEBUG:teuthology.orchestra.run.smithi018:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:06:12.069 DEBUG:teuthology.orchestra.run.smithi070:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:06:12.135 INFO:teuthology.orchestra.run.smithi018.stdout:check_obsoletes = 1 2024-03-29T11:06:12.136 INFO:teuthology.orchestra.run.smithi070.stdout:check_obsoletes = 1 2024-03-29T11:06:12.137 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum clean all 2024-03-29T11:06:12.138 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum clean all 2024-03-29T11:06:12.372 INFO:teuthology.orchestra.run.smithi070.stdout:74 files removed 2024-03-29T11:06:12.381 INFO:teuthology.orchestra.run.smithi018.stdout:74 files removed 2024-03-29T11:06:12.399 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-radosgw 2024-03-29T11:06:12.407 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-radosgw 2024-03-29T11:06:13.024 INFO:teuthology.orchestra.run.smithi018.stdout:ceph packages for x86_64 265 kB/s | 82 kB 00:00 2024-03-29T11:06:13.088 INFO:teuthology.orchestra.run.smithi070.stdout:ceph packages for x86_64 211 kB/s | 82 kB 00:00 2024-03-29T11:06:13.399 INFO:teuthology.orchestra.run.smithi018.stdout:ceph noarch packages 31 kB/s | 10 kB 00:00 2024-03-29T11:06:13.439 INFO:teuthology.orchestra.run.smithi070.stdout:ceph noarch packages 33 kB/s | 10 kB 00:00 2024-03-29T11:06:13.817 INFO:teuthology.orchestra.run.smithi018.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-03-29T11:06:13.848 INFO:teuthology.orchestra.run.smithi070.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2024-03-29T11:06:14.219 INFO:teuthology.orchestra.run.smithi018.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-29T11:06:14.567 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS-9 - AppStream 9.8 MB/s | 6.7 MB 00:00 2024-03-29T11:06:15.728 INFO:teuthology.orchestra.run.smithi018.stdout:CentOS-9 - BaseOS 5.8 MB/s | 2.0 MB 00:00 2024-03-29T11:06:16.075 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS-9 - BaseOS 5.1 MB/s | 2.0 MB 00:00 2024-03-29T11:06:16.660 INFO:teuthology.orchestra.run.smithi018.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-03-29T11:06:17.051 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-29T11:06:18.594 INFO:teuthology.orchestra.run.smithi018.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-03-29T11:06:19.238 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - AppStream 17 MB/s | 19 MB 00:01 2024-03-29T11:06:23.853 INFO:teuthology.orchestra.run.smithi018.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 6.1 MB 00:01 2024-03-29T11:06:25.187 INFO:teuthology.orchestra.run.smithi018.stdout:CentOS Stream 9 - Extras packages 102 kB/s | 16 kB 00:00 2024-03-29T11:06:25.235 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - CRB 2.5 MB/s | 6.1 MB 00:02 2024-03-29T11:06:26.531 INFO:teuthology.orchestra.run.smithi018.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-29T11:06:26.586 INFO:teuthology.orchestra.run.smithi070.stdout:CentOS Stream 9 - Extras packages 82 kB/s | 16 kB 00:00 2024-03-29T11:06:28.423 INFO:teuthology.orchestra.run.smithi070.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-29T11:06:31.706 INFO:teuthology.orchestra.run.smithi018.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-29T11:06:33.387 INFO:teuthology.orchestra.run.smithi070.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2024-03-29T11:06:34.408 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-29T11:06:34.420 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-29T11:06:34.420 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repository Size 2024-03-29T11:06:34.420 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-29T11:06:34.420 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-29T11:06:34.421 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:06:34.421 INFO:teuthology.orchestra.run.smithi018.stdout:Upgrading: 2024-03-29T11:06:34.421 INFO:teuthology.orchestra.run.smithi018.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:06:34.421 INFO:teuthology.orchestra.run.smithi018.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:06:34.421 INFO:teuthology.orchestra.run.smithi018.stdout:Installing dependencies: 2024-03-29T11:06:34.421 INFO:teuthology.orchestra.run.smithi018.stdout: boost-program-options 2024-03-29T11:06:34.421 INFO:teuthology.orchestra.run.smithi018.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:06:34.421 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:06:34.421 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:06:34.421 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:06:34.421 INFO:teuthology.orchestra.run.smithi018.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:06:34.421 INFO:teuthology.orchestra.run.smithi018.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:06:34.421 INFO:teuthology.orchestra.run.smithi018.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:06:34.421 INFO:teuthology.orchestra.run.smithi018.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:06:34.422 INFO:teuthology.orchestra.run.smithi018.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:06:34.422 INFO:teuthology.orchestra.run.smithi018.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:06:34.422 INFO:teuthology.orchestra.run.smithi018.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:06:34.422 INFO:teuthology.orchestra.run.smithi018.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:06:34.422 INFO:teuthology.orchestra.run.smithi018.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:06:34.422 INFO:teuthology.orchestra.run.smithi018.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:06:34.422 INFO:teuthology.orchestra.run.smithi018.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:06:34.422 INFO:teuthology.orchestra.run.smithi018.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:06:34.422 INFO:teuthology.orchestra.run.smithi018.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:06:34.422 INFO:teuthology.orchestra.run.smithi018.stdout: python3-ceph-argparse 2024-03-29T11:06:34.422 INFO:teuthology.orchestra.run.smithi018.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:06:34.422 INFO:teuthology.orchestra.run.smithi018.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:06:34.422 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:06:34.423 INFO:teuthology.orchestra.run.smithi018.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:06:34.423 INFO:teuthology.orchestra.run.smithi018.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:06:34.423 INFO:teuthology.orchestra.run.smithi018.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:06:34.423 INFO:teuthology.orchestra.run.smithi018.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:06:34.423 INFO:teuthology.orchestra.run.smithi018.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:06:34.423 INFO:teuthology.orchestra.run.smithi018.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:06:34.423 INFO:teuthology.orchestra.run.smithi018.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:06:34.423 INFO:teuthology.orchestra.run.smithi018.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:06:34.423 INFO:teuthology.orchestra.run.smithi018.stdout:Installing weak dependencies: 2024-03-29T11:06:34.423 INFO:teuthology.orchestra.run.smithi018.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:06:34.423 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:06:34.423 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-29T11:06:34.423 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-29T11:06:34.423 INFO:teuthology.orchestra.run.smithi018.stdout:Install 30 Packages 2024-03-29T11:06:34.423 INFO:teuthology.orchestra.run.smithi018.stdout:Upgrade 2 Packages 2024-03-29T11:06:34.423 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:06:34.424 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 58 M 2024-03-29T11:06:34.424 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-29T11:06:35.531 INFO:teuthology.orchestra.run.smithi018.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 5.6 MB/s | 5.3 MB 00:00 2024-03-29T11:06:35.606 INFO:teuthology.orchestra.run.smithi018.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 336 kB/s | 25 kB 00:00 2024-03-29T11:06:35.765 INFO:teuthology.orchestra.run.smithi018.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 4.6 MB/s | 742 kB 00:00 2024-03-29T11:06:35.865 INFO:teuthology.orchestra.run.smithi018.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 4.9 MB/s | 498 kB 00:00 2024-03-29T11:06:35.965 INFO:teuthology.orchestra.run.smithi018.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 7.0 MB/s | 9.7 MB 00:01 2024-03-29T11:06:35.973 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-03-29T11:06:35.979 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-03-29T11:06:35.979 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-03-29T11:06:35.979 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-03-29T11:06:35.979 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-03-29T11:06:35.979 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:06:35.979 INFO:teuthology.orchestra.run.smithi070.stdout:Upgrading: 2024-03-29T11:06:35.980 INFO:teuthology.orchestra.run.smithi070.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:06:35.980 INFO:teuthology.orchestra.run.smithi070.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:06:35.980 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-03-29T11:06:35.980 INFO:teuthology.orchestra.run.smithi070.stdout: boost-program-options 2024-03-29T11:06:35.980 INFO:teuthology.orchestra.run.smithi070.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:06:35.980 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:06:35.980 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:06:35.980 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:06:35.980 INFO:teuthology.orchestra.run.smithi070.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:06:35.980 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:06:35.980 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:06:35.980 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:06:35.980 INFO:teuthology.orchestra.run.smithi070.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:06:35.980 INFO:teuthology.orchestra.run.smithi070.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:06:35.981 INFO:teuthology.orchestra.run.smithi070.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:06:35.981 INFO:teuthology.orchestra.run.smithi070.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:06:35.981 INFO:teuthology.orchestra.run.smithi070.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:06:35.981 INFO:teuthology.orchestra.run.smithi070.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:06:35.981 INFO:teuthology.orchestra.run.smithi070.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:06:35.981 INFO:teuthology.orchestra.run.smithi070.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:06:35.981 INFO:teuthology.orchestra.run.smithi070.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:06:35.981 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-argparse 2024-03-29T11:06:35.981 INFO:teuthology.orchestra.run.smithi070.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:06:35.981 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:06:35.981 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:06:35.981 INFO:teuthology.orchestra.run.smithi070.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:06:35.981 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:06:35.981 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:06:35.981 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:06:35.981 INFO:teuthology.orchestra.run.smithi070.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:06:35.982 INFO:teuthology.orchestra.run.smithi070.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:06:35.982 INFO:teuthology.orchestra.run.smithi070.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:06:35.982 INFO:teuthology.orchestra.run.smithi070.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:06:35.982 INFO:teuthology.orchestra.run.smithi070.stdout:Installing weak dependencies: 2024-03-29T11:06:35.982 INFO:teuthology.orchestra.run.smithi070.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:06:35.982 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:06:35.982 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-03-29T11:06:35.982 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-03-29T11:06:35.982 INFO:teuthology.orchestra.run.smithi070.stdout:Install 30 Packages 2024-03-29T11:06:35.982 INFO:teuthology.orchestra.run.smithi070.stdout:Upgrade 2 Packages 2024-03-29T11:06:35.982 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:06:35.983 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 58 M 2024-03-29T11:06:35.983 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-03-29T11:06:36.040 INFO:teuthology.orchestra.run.smithi018.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 604 kB/s | 45 kB 00:00 2024-03-29T11:06:36.124 INFO:teuthology.orchestra.run.smithi018.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.5 MB/s | 131 kB 00:00 2024-03-29T11:06:36.208 INFO:teuthology.orchestra.run.smithi018.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 1.9 MB/s | 162 kB 00:00 2024-03-29T11:06:36.300 INFO:teuthology.orchestra.run.smithi018.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 3.4 MB/s | 320 kB 00:00 2024-03-29T11:06:36.383 INFO:teuthology.orchestra.run.smithi018.stdout:(10/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 9.9 MB/s | 5.1 MB 00:00 2024-03-29T11:06:36.408 INFO:teuthology.orchestra.run.smithi018.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 2.7 MB/s | 298 kB 00:00 2024-03-29T11:06:36.467 INFO:teuthology.orchestra.run.smithi018.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.2 MB/s | 99 kB 00:00 2024-03-29T11:06:36.526 INFO:teuthology.orchestra.run.smithi018.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 904 kB/s | 106 kB 00:00 2024-03-29T11:06:36.551 INFO:teuthology.orchestra.run.smithi018.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 159 kB 00:00 2024-03-29T11:06:36.576 INFO:teuthology.orchestra.run.smithi018.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 886 kB/s | 44 kB 00:00 2024-03-29T11:06:36.610 INFO:teuthology.orchestra.run.smithi018.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-29T11:06:36.635 INFO:teuthology.orchestra.run.smithi018.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2024-03-29T11:06:36.660 INFO:teuthology.orchestra.run.smithi018.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 827 kB/s | 41 kB 00:00 2024-03-29T11:06:36.686 INFO:teuthology.orchestra.run.smithi018.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-03-29T11:06:36.711 INFO:teuthology.orchestra.run.smithi018.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 697 kB/s | 35 kB 00:00 2024-03-29T11:06:36.736 INFO:teuthology.orchestra.run.smithi018.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 922 kB/s | 46 kB 00:00 2024-03-29T11:06:36.928 INFO:teuthology.orchestra.run.smithi018.stdout:(22/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 8.5 MB/s | 20 MB 00:02 2024-03-29T11:06:37.004 INFO:teuthology.orchestra.run.smithi070.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 5.7 MB/s | 5.3 MB 00:00 2024-03-29T11:06:37.114 INFO:teuthology.orchestra.run.smithi070.stdout:(2/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 9.3 MB/s | 9.7 MB 00:01 2024-03-29T11:06:37.129 INFO:teuthology.orchestra.run.smithi018.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 175 kB/s | 26 kB 00:00 2024-03-29T11:06:37.146 INFO:teuthology.orchestra.run.smithi070.stdout:(3/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 177 kB/s | 25 kB 00:00 2024-03-29T11:06:37.154 INFO:teuthology.orchestra.run.smithi018.stdout:(24/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 697 kB/s | 309 kB 00:00 2024-03-29T11:06:37.180 INFO:teuthology.orchestra.run.smithi070.stdout:(4/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 11 MB/s | 742 kB 00:00 2024-03-29T11:06:37.196 INFO:teuthology.orchestra.run.smithi018.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 730 kB/s | 49 kB 00:00 2024-03-29T11:06:37.214 INFO:teuthology.orchestra.run.smithi070.stdout:(5/32): libradosstriper1-19.0.0-1439.g5dcb190d. 7.3 MB/s | 498 kB 00:00 2024-03-29T11:06:37.221 INFO:teuthology.orchestra.run.smithi018.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-29T11:06:37.289 INFO:teuthology.orchestra.run.smithi070.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 603 kB/s | 45 kB 00:00 2024-03-29T11:06:37.330 INFO:teuthology.orchestra.run.smithi018.stdout:(27/32): libarrow-9.0.0-10.el9.x86_64.rpm 7.4 MB/s | 4.4 MB 00:00 2024-03-29T11:06:37.372 INFO:teuthology.orchestra.run.smithi070.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.5 MB/s | 131 kB 00:00 2024-03-29T11:06:37.380 INFO:teuthology.orchestra.run.smithi018.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.5 MB/s | 838 kB 00:00 2024-03-29T11:06:37.405 INFO:teuthology.orchestra.run.smithi018.stdout:(29/32): re2-20211101-3.el9.x86_64.rpm 1.0 MB/s | 192 kB 00:00 2024-03-29T11:06:37.447 INFO:teuthology.orchestra.run.smithi018.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-03-29T11:06:37.456 INFO:teuthology.orchestra.run.smithi070.stdout:(8/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 19 MB/s | 5.1 MB 00:00 2024-03-29T11:06:37.473 INFO:teuthology.orchestra.run.smithi070.stdout:(9/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 1.6 MB/s | 162 kB 00:00 2024-03-29T11:06:37.506 INFO:teuthology.orchestra.run.smithi070.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 6.2 MB/s | 320 kB 00:00 2024-03-29T11:06:37.532 INFO:teuthology.orchestra.run.smithi070.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 5.0 MB/s | 298 kB 00:00 2024-03-29T11:06:37.565 INFO:teuthology.orchestra.run.smithi070.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.7 MB/s | 99 kB 00:00 2024-03-29T11:06:37.699 INFO:teuthology.orchestra.run.smithi070.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 633 kB/s | 106 kB 00:00 2024-03-29T11:06:37.724 INFO:teuthology.orchestra.run.smithi070.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 159 kB 00:00 2024-03-29T11:06:37.749 INFO:teuthology.orchestra.run.smithi070.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 883 kB/s | 44 kB 00:00 2024-03-29T11:06:37.789 INFO:teuthology.orchestra.run.smithi018.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 8.8 MB/s | 3.4 MB 00:00 2024-03-29T11:06:37.808 INFO:teuthology.orchestra.run.smithi070.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.7 MB/s | 661 kB 00:00 2024-03-29T11:06:37.833 INFO:teuthology.orchestra.run.smithi070.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2024-03-29T11:06:37.839 INFO:teuthology.orchestra.run.smithi018.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 7.3 MB/s | 3.1 MB 00:00 2024-03-29T11:06:37.846 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:06:37.846 INFO:teuthology.orchestra.run.smithi018.stdout:Total 17 MB/s | 58 MB 00:03 2024-03-29T11:06:37.846 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-29T11:06:37.858 INFO:teuthology.orchestra.run.smithi070.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 829 kB/s | 41 kB 00:00 2024-03-29T11:06:37.884 INFO:teuthology.orchestra.run.smithi070.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-03-29T11:06:37.889 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-29T11:06:37.889 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-29T11:06:37.909 INFO:teuthology.orchestra.run.smithi070.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 694 kB/s | 35 kB 00:00 2024-03-29T11:06:37.934 INFO:teuthology.orchestra.run.smithi070.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 925 kB/s | 46 kB 00:00 2024-03-29T11:06:38.102 INFO:teuthology.orchestra.run.smithi070.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.6 MB/s | 309 kB 00:00 2024-03-29T11:06:38.143 INFO:teuthology.orchestra.run.smithi070.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 623 kB/s | 26 kB 00:00 2024-03-29T11:06:38.186 INFO:teuthology.orchestra.run.smithi070.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-03-29T11:06:38.260 INFO:teuthology.orchestra.run.smithi070.stdout:(25/32): libarrow-9.0.0-10.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-03-29T11:06:38.278 INFO:teuthology.orchestra.run.smithi070.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 734 kB/s | 67 kB 00:00 2024-03-29T11:06:38.311 INFO:teuthology.orchestra.run.smithi070.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 16 MB/s | 838 kB 00:00 2024-03-29T11:06:38.336 INFO:teuthology.orchestra.run.smithi070.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-03-29T11:06:38.378 INFO:teuthology.orchestra.run.smithi070.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 24 MB/s | 1.6 MB 00:00 2024-03-29T11:06:38.379 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-29T11:06:38.380 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-29T11:06:38.728 INFO:teuthology.orchestra.run.smithi070.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 8.9 MB/s | 3.1 MB 00:00 2024-03-29T11:06:38.787 INFO:teuthology.orchestra.run.smithi070.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 7.5 MB/s | 3.4 MB 00:00 2024-03-29T11:06:39.354 INFO:teuthology.orchestra.run.smithi070.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 6.0 MB/s | 20 MB 00:03 2024-03-29T11:06:39.358 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:06:39.358 INFO:teuthology.orchestra.run.smithi070.stdout:Total 17 MB/s | 58 MB 00:03 2024-03-29T11:06:39.359 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-03-29T11:06:39.391 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-03-29T11:06:39.391 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-03-29T11:06:39.861 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-03-29T11:06:39.861 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-03-29T11:06:39.892 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-29T11:06:39.971 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:06:40.010 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:06:40.036 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:06:40.058 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:06:40.076 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:06:40.217 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:06:40.278 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:06:40.674 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:06:40.712 INFO:teuthology.orchestra.run.smithi018.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:06:40.805 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:06:40.894 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:06:40.922 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:06:40.984 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:06:41.005 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:06:41.041 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:06:41.110 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:06:41.132 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:06:41.196 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:06:41.231 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:06:41.343 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:06:41.343 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-03-29T11:06:41.374 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:06:41.415 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:06:41.439 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:06:41.447 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:06:41.475 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:06:41.475 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:06:41.500 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:06:41.509 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:06:41.523 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:06:41.540 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:06:41.681 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:06:41.736 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:06:41.844 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:06:41.872 INFO:teuthology.orchestra.run.smithi018.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:06:41.962 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:06:41.993 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:06:42.042 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:06:42.067 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:06:42.144 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:06:42.235 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:06:42.325 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:06:42.353 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:06:42.424 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:06:42.453 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:06:42.497 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:06:42.530 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:06:42.557 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:06:42.566 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:06:42.597 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:06:42.620 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:06:42.669 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:06:42.691 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:06:42.704 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:06:42.718 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:06:42.815 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:06:42.847 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:06:42.887 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:06:42.920 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:06:42.949 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:06:42.982 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:06:43.292 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:06:43.319 INFO:teuthology.orchestra.run.smithi070.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:06:43.392 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:06:43.424 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:06:43.464 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:06:43.918 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:06:43.938 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:06:44.010 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:06:44.069 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:06:44.099 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:06:44.983 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:06:45.058 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:06:45.659 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:06:45.701 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:06:45.777 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:06:45.777 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:06:45.777 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:06:45.778 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:06:45.802 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:06:46.302 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:06:46.336 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:06:46.883 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:06:46.923 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:06:47.015 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:06:47.016 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:06:47.016 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:06:47.016 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:06:47.058 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:06:56.129 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:06:56.130 INFO:teuthology.orchestra.run.smithi018.stdout:skipping the directory /sys 2024-03-29T11:06:56.130 INFO:teuthology.orchestra.run.smithi018.stdout:skipping the directory /proc 2024-03-29T11:06:56.130 INFO:teuthology.orchestra.run.smithi018.stdout:skipping the directory /mnt 2024-03-29T11:06:56.130 INFO:teuthology.orchestra.run.smithi018.stdout:skipping the directory /var/tmp 2024-03-29T11:06:56.130 INFO:teuthology.orchestra.run.smithi018.stdout:skipping the directory /home 2024-03-29T11:06:56.130 INFO:teuthology.orchestra.run.smithi018.stdout:skipping the directory /root 2024-03-29T11:06:56.130 INFO:teuthology.orchestra.run.smithi018.stdout:skipping the directory /tmp 2024-03-29T11:06:56.130 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:06:56.412 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:06:56.412 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /sys 2024-03-29T11:06:56.412 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /proc 2024-03-29T11:06:56.412 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /mnt 2024-03-29T11:06:56.412 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /var/tmp 2024-03-29T11:06:56.413 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /home 2024-03-29T11:06:56.413 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /root 2024-03-29T11:06:56.413 INFO:teuthology.orchestra.run.smithi070.stdout:skipping the directory /tmp 2024-03-29T11:06:56.413 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:06:57.143 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:06:57.168 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:06:57.168 INFO:teuthology.orchestra.run.smithi018.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:06:57.168 INFO:teuthology.orchestra.run.smithi018.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:06:57.168 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:06:57.168 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:06:57.168 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:06:57.170 INFO:teuthology.orchestra.run.smithi018.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:06:57.254 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:06:57.254 INFO:teuthology.orchestra.run.smithi018.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:06:57.332 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:06:57.356 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:06:57.356 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:06:57.356 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:06:57.357 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:06:57.357 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:06:57.357 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:06:57.358 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:06:57.442 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:06:57.443 INFO:teuthology.orchestra.run.smithi070.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:06:58.398 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:06:58.398 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:06:58.398 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:06:58.398 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:06:58.398 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:06:58.398 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:06:58.398 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:06:58.399 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:06:58.399 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:06:58.399 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:06:58.399 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:06:58.399 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:06:58.399 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:06:58.399 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:06:58.399 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:06:58.399 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:06:58.399 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:06:58.399 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:06:58.399 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:06:58.399 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:06:58.399 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:06:58.400 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:06:58.400 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:06:58.400 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:06:58.400 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:06:58.400 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:06:58.400 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:06:58.400 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:06:58.400 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:06:58.400 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:06:58.400 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:06:58.400 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:06:58.400 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:06:58.400 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:06:58.552 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:06:58.553 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:06:58.553 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:06:58.553 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:06:58.553 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:06:58.553 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:06:58.553 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:06:58.553 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:06:58.553 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:06:58.553 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:06:58.553 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:06:58.553 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:06:58.553 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:06:58.553 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:06:58.554 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:06:58.554 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:06:58.554 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:06:58.554 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:06:58.554 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:06:58.554 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:06:58.554 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:06:58.554 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:06:58.554 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:06:58.554 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:06:58.554 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:06:58.554 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:06:58.554 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:06:58.555 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:06:58.555 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:06:58.555 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:06:58.555 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:06:58.555 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:06:58.555 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:06:58.555 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:06:59.588 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:06:59.588 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:06:59.588 INFO:teuthology.orchestra.run.smithi018.stdout:Upgraded: 2024-03-29T11:06:59.588 INFO:teuthology.orchestra.run.smithi018.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:59.588 INFO:teuthology.orchestra.run.smithi018.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:59.589 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-29T11:06:59.589 INFO:teuthology.orchestra.run.smithi018.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:06:59.589 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:59.589 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:59.589 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:59.589 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:59.589 INFO:teuthology.orchestra.run.smithi018.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:06:59.589 INFO:teuthology.orchestra.run.smithi018.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:06:59.589 INFO:teuthology.orchestra.run.smithi018.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:06:59.589 INFO:teuthology.orchestra.run.smithi018.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:59.589 INFO:teuthology.orchestra.run.smithi018.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:06:59.589 INFO:teuthology.orchestra.run.smithi018.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:06:59.589 INFO:teuthology.orchestra.run.smithi018.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:06:59.589 INFO:teuthology.orchestra.run.smithi018.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:59.590 INFO:teuthology.orchestra.run.smithi018.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:06:59.590 INFO:teuthology.orchestra.run.smithi018.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:59.590 INFO:teuthology.orchestra.run.smithi018.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:06:59.590 INFO:teuthology.orchestra.run.smithi018.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:06:59.590 INFO:teuthology.orchestra.run.smithi018.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:06:59.590 INFO:teuthology.orchestra.run.smithi018.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:59.590 INFO:teuthology.orchestra.run.smithi018.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:59.590 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:59.590 INFO:teuthology.orchestra.run.smithi018.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:06:59.590 INFO:teuthology.orchestra.run.smithi018.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:59.590 INFO:teuthology.orchestra.run.smithi018.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:59.590 INFO:teuthology.orchestra.run.smithi018.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:59.590 INFO:teuthology.orchestra.run.smithi018.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:06:59.590 INFO:teuthology.orchestra.run.smithi018.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:06:59.591 INFO:teuthology.orchestra.run.smithi018.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:06:59.591 INFO:teuthology.orchestra.run.smithi018.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:06:59.591 INFO:teuthology.orchestra.run.smithi018.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:06:59.591 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:06:59.591 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-29T11:06:59.666 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:06:59.666 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:06:59.666 INFO:teuthology.orchestra.run.smithi070.stdout:Upgraded: 2024-03-29T11:06:59.667 INFO:teuthology.orchestra.run.smithi070.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:59.667 INFO:teuthology.orchestra.run.smithi070.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:59.667 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-03-29T11:06:59.667 INFO:teuthology.orchestra.run.smithi070.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:06:59.667 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:59.667 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:59.667 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:59.667 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:59.667 INFO:teuthology.orchestra.run.smithi070.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:06:59.667 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:06:59.667 INFO:teuthology.orchestra.run.smithi070.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:06:59.667 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:59.667 INFO:teuthology.orchestra.run.smithi070.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:06:59.667 INFO:teuthology.orchestra.run.smithi070.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:06:59.668 INFO:teuthology.orchestra.run.smithi070.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:06:59.668 INFO:teuthology.orchestra.run.smithi070.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:59.668 INFO:teuthology.orchestra.run.smithi070.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:06:59.668 INFO:teuthology.orchestra.run.smithi070.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:59.668 INFO:teuthology.orchestra.run.smithi070.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:06:59.668 INFO:teuthology.orchestra.run.smithi070.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:06:59.669 INFO:teuthology.orchestra.run.smithi070.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:06:59.669 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:59.669 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:59.669 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:59.669 INFO:teuthology.orchestra.run.smithi070.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:06:59.669 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:59.669 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:59.669 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:06:59.669 INFO:teuthology.orchestra.run.smithi070.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:06:59.669 INFO:teuthology.orchestra.run.smithi070.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:06:59.669 INFO:teuthology.orchestra.run.smithi070.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:06:59.669 INFO:teuthology.orchestra.run.smithi070.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:06:59.669 INFO:teuthology.orchestra.run.smithi070.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:06:59.670 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:06:59.670 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-03-29T11:06:59.897 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-test 2024-03-29T11:07:00.003 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-test 2024-03-29T11:07:00.471 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:00:29 ago on Fri 29 Mar 2024 11:06:31 AM UTC. 2024-03-29T11:07:00.554 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:00:27 ago on Fri 29 Mar 2024 11:06:33 AM UTC. 2024-03-29T11:07:00.627 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-29T11:07:00.628 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-29T11:07:00.628 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repository Size 2024-03-29T11:07:00.628 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-29T11:07:00.629 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-29T11:07:00.629 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:07:00.629 INFO:teuthology.orchestra.run.smithi018.stdout:Installing dependencies: 2024-03-29T11:07:00.629 INFO:teuthology.orchestra.run.smithi018.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:07:00.629 INFO:teuthology.orchestra.run.smithi018.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:07:00.629 INFO:teuthology.orchestra.run.smithi018.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:07:00.629 INFO:teuthology.orchestra.run.smithi018.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:07:00.629 INFO:teuthology.orchestra.run.smithi018.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:07:00.629 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:07:00.629 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-29T11:07:00.629 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-29T11:07:00.629 INFO:teuthology.orchestra.run.smithi018.stdout:Install 6 Packages 2024-03-29T11:07:00.629 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:07:00.630 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 48 M 2024-03-29T11:07:00.630 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 204 M 2024-03-29T11:07:00.630 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-29T11:07:00.712 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-03-29T11:07:00.713 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-03-29T11:07:00.713 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-03-29T11:07:00.713 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-03-29T11:07:00.713 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-03-29T11:07:00.713 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:07:00.713 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-03-29T11:07:00.713 INFO:teuthology.orchestra.run.smithi070.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:07:00.713 INFO:teuthology.orchestra.run.smithi070.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:07:00.713 INFO:teuthology.orchestra.run.smithi070.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:07:00.713 INFO:teuthology.orchestra.run.smithi070.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:07:00.713 INFO:teuthology.orchestra.run.smithi070.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:07:00.714 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:07:00.714 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-03-29T11:07:00.714 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-03-29T11:07:00.714 INFO:teuthology.orchestra.run.smithi070.stdout:Install 6 Packages 2024-03-29T11:07:00.714 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:07:00.714 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 48 M 2024-03-29T11:07:00.714 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 204 M 2024-03-29T11:07:00.715 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-03-29T11:07:00.994 INFO:teuthology.orchestra.run.smithi070.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-29T11:07:01.060 INFO:teuthology.orchestra.run.smithi018.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-29T11:07:01.061 INFO:teuthology.orchestra.run.smithi070.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 4.4 MB/s | 300 kB 00:00 2024-03-29T11:07:01.086 INFO:teuthology.orchestra.run.smithi070.stdout:(3/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 628 kB/s | 163 kB 00:00 2024-03-29T11:07:01.111 INFO:teuthology.orchestra.run.smithi070.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-29T11:07:01.135 INFO:teuthology.orchestra.run.smithi018.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 3.9 MB/s | 300 kB 00:00 2024-03-29T11:07:01.160 INFO:teuthology.orchestra.run.smithi018.stdout:(3/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 633 kB/s | 163 kB 00:00 2024-03-29T11:07:01.186 INFO:teuthology.orchestra.run.smithi018.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-29T11:07:01.394 INFO:teuthology.orchestra.run.smithi018.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 801 kB/s | 187 kB 00:00 2024-03-29T11:07:01.612 INFO:teuthology.orchestra.run.smithi070.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 356 kB/s | 187 kB 00:00 2024-03-29T11:07:04.886 INFO:teuthology.orchestra.run.smithi070.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 12 MB/s | 47 MB 00:04 2024-03-29T11:07:04.890 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:04.890 INFO:teuthology.orchestra.run.smithi070.stdout:Total 12 MB/s | 48 MB 00:04 2024-03-29T11:07:04.904 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-03-29T11:07:04.919 INFO:teuthology.orchestra.run.smithi018.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 12 MB/s | 47 MB 00:04 2024-03-29T11:07:04.922 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:04.923 INFO:teuthology.orchestra.run.smithi018.stdout:Total 11 MB/s | 48 MB 00:04 2024-03-29T11:07:04.925 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-03-29T11:07:04.926 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-03-29T11:07:04.937 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-29T11:07:04.961 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-29T11:07:04.961 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-29T11:07:05.307 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-03-29T11:07:05.307 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-03-29T11:07:05.357 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-29T11:07:05.357 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-29T11:07:05.905 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-03-29T11:07:05.955 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-29T11:07:05.978 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:07:06.012 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:07:06.027 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:07:06.050 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:07:06.066 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:07:06.090 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:07:06.112 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:07:06.136 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:07:06.139 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:07:06.164 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:07:10.668 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:07:10.747 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:07:10.918 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:07:10.999 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:07:13.225 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:07:13.226 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:07:13.226 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:07:13.226 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:07:13.226 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:07:13.226 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:07:13.412 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:07:13.412 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:07:13.412 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:07:13.412 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:07:13.413 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:07:13.413 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:07:13.612 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:07:13.612 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:07:13.613 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-03-29T11:07:13.613 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:07:13.613 INFO:teuthology.orchestra.run.smithi070.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:07:13.613 INFO:teuthology.orchestra.run.smithi070.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:07:13.613 INFO:teuthology.orchestra.run.smithi070.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:07:13.613 INFO:teuthology.orchestra.run.smithi070.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:07:13.613 INFO:teuthology.orchestra.run.smithi070.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:07:13.613 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:07:13.613 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-03-29T11:07:13.814 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:07:13.814 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:07:13.815 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-29T11:07:13.815 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:07:13.815 INFO:teuthology.orchestra.run.smithi018.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:07:13.815 INFO:teuthology.orchestra.run.smithi018.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:07:13.815 INFO:teuthology.orchestra.run.smithi018.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:07:13.815 INFO:teuthology.orchestra.run.smithi018.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:07:13.815 INFO:teuthology.orchestra.run.smithi018.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:07:13.815 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:07:13.815 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-29T11:07:13.921 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph 2024-03-29T11:07:14.114 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph 2024-03-29T11:07:14.517 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:00:41 ago on Fri 29 Mar 2024 11:06:33 AM UTC. 2024-03-29T11:07:14.675 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-03-29T11:07:14.677 INFO:teuthology.orchestra.run.smithi070.stdout:========================================================================================= 2024-03-29T11:07:14.677 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-03-29T11:07:14.677 INFO:teuthology.orchestra.run.smithi070.stdout:========================================================================================= 2024-03-29T11:07:14.677 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-03-29T11:07:14.677 INFO:teuthology.orchestra.run.smithi070.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:07:14.677 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-03-29T11:07:14.677 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:07:14.677 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:07:14.677 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:07:14.678 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:07:14.678 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:07:14.678 INFO:teuthology.orchestra.run.smithi070.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:07:14.678 INFO:teuthology.orchestra.run.smithi070.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:07:14.678 INFO:teuthology.orchestra.run.smithi070.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:07:14.678 INFO:teuthology.orchestra.run.smithi070.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:07:14.678 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:07:14.678 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:07:14.678 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:07:14.678 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:07:14.678 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:07:14.678 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:07:14.679 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:07:14.679 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:07:14.679 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:07:14.679 INFO:teuthology.orchestra.run.smithi070.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:07:14.679 INFO:teuthology.orchestra.run.smithi070.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:07:14.679 INFO:teuthology.orchestra.run.smithi070.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:07:14.679 INFO:teuthology.orchestra.run.smithi070.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:07:14.679 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:07:14.679 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:07:14.679 INFO:teuthology.orchestra.run.smithi070.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:07:14.679 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:07:14.679 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:07:14.679 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:07:14.679 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:07:14.680 INFO:teuthology.orchestra.run.smithi070.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:07:14.680 INFO:teuthology.orchestra.run.smithi070.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:07:14.680 INFO:teuthology.orchestra.run.smithi070.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:07:14.680 INFO:teuthology.orchestra.run.smithi070.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:07:14.680 INFO:teuthology.orchestra.run.smithi070.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:07:14.680 INFO:teuthology.orchestra.run.smithi070.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:07:14.680 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:07:14.680 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-03-29T11:07:14.680 INFO:teuthology.orchestra.run.smithi070.stdout:========================================================================================= 2024-03-29T11:07:14.680 INFO:teuthology.orchestra.run.smithi070.stdout:Install 36 Packages 2024-03-29T11:07:14.680 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:07:14.681 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 30 M 2024-03-29T11:07:14.681 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 107 M 2024-03-29T11:07:14.681 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-03-29T11:07:14.686 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:00:43 ago on Fri 29 Mar 2024 11:06:31 AM UTC. 2024-03-29T11:07:14.847 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-29T11:07:14.849 INFO:teuthology.orchestra.run.smithi018.stdout:========================================================================================= 2024-03-29T11:07:14.850 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repository Size 2024-03-29T11:07:14.850 INFO:teuthology.orchestra.run.smithi018.stdout:========================================================================================= 2024-03-29T11:07:14.850 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-29T11:07:14.850 INFO:teuthology.orchestra.run.smithi018.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:07:14.850 INFO:teuthology.orchestra.run.smithi018.stdout:Installing dependencies: 2024-03-29T11:07:14.850 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:07:14.851 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:07:14.851 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:07:14.851 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:07:14.851 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:07:14.851 INFO:teuthology.orchestra.run.smithi018.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:07:14.851 INFO:teuthology.orchestra.run.smithi018.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:07:14.852 INFO:teuthology.orchestra.run.smithi018.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:07:14.852 INFO:teuthology.orchestra.run.smithi018.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:07:14.852 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:07:14.852 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:07:14.852 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:07:14.852 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:07:14.853 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:07:14.853 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:07:14.853 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:07:14.853 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:07:14.853 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:07:14.853 INFO:teuthology.orchestra.run.smithi018.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:07:14.854 INFO:teuthology.orchestra.run.smithi018.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:07:14.854 INFO:teuthology.orchestra.run.smithi018.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:07:14.854 INFO:teuthology.orchestra.run.smithi018.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:07:14.854 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:07:14.854 INFO:teuthology.orchestra.run.smithi018.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:07:14.854 INFO:teuthology.orchestra.run.smithi018.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:07:14.855 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:07:14.855 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:07:14.855 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:07:14.855 INFO:teuthology.orchestra.run.smithi018.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:07:14.855 INFO:teuthology.orchestra.run.smithi018.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:07:14.855 INFO:teuthology.orchestra.run.smithi018.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:07:14.855 INFO:teuthology.orchestra.run.smithi018.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:07:14.856 INFO:teuthology.orchestra.run.smithi018.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:07:14.856 INFO:teuthology.orchestra.run.smithi018.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:07:14.856 INFO:teuthology.orchestra.run.smithi018.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:07:14.856 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:07:14.856 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-29T11:07:14.856 INFO:teuthology.orchestra.run.smithi018.stdout:========================================================================================= 2024-03-29T11:07:14.857 INFO:teuthology.orchestra.run.smithi018.stdout:Install 36 Packages 2024-03-29T11:07:14.857 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:07:14.858 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 30 M 2024-03-29T11:07:14.858 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 107 M 2024-03-29T11:07:14.858 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-29T11:07:15.313 INFO:teuthology.orchestra.run.smithi070.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 40 kB/s | 6.6 kB 00:00 2024-03-29T11:07:15.456 INFO:teuthology.orchestra.run.smithi018.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 29 kB/s | 6.6 kB 00:00 2024-03-29T11:07:15.731 INFO:teuthology.orchestra.run.smithi018.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 4.2 MB/s | 2.1 MB 00:00 2024-03-29T11:07:15.813 INFO:teuthology.orchestra.run.smithi070.stdout:(2/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 9.4 MB/s | 4.7 MB 00:00 2024-03-29T11:07:15.864 INFO:teuthology.orchestra.run.smithi070.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 3.0 MB/s | 2.1 MB 00:00 2024-03-29T11:07:15.964 INFO:teuthology.orchestra.run.smithi070.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 2.4 MB/s | 247 kB 00:00 2024-03-29T11:07:16.081 INFO:teuthology.orchestra.run.smithi070.stdout:(5/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 1.6 MB/s | 1.5 MB 00:00 2024-03-29T11:07:16.106 INFO:teuthology.orchestra.run.smithi070.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-03-29T11:07:16.139 INFO:teuthology.orchestra.run.smithi070.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-03-29T11:07:16.165 INFO:teuthology.orchestra.run.smithi070.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 538 kB/s | 32 kB 00:00 2024-03-29T11:07:16.202 INFO:teuthology.orchestra.run.smithi070.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 807 kB/s | 47 kB 00:00 2024-03-29T11:07:16.224 INFO:teuthology.orchestra.run.smithi070.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 660 kB/s | 39 kB 00:00 2024-03-29T11:07:16.299 INFO:teuthology.orchestra.run.smithi070.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.3 MB/s | 241 kB 00:00 2024-03-29T11:07:16.323 INFO:teuthology.orchestra.run.smithi018.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 1.4 MB/s | 1.5 MB 00:01 2024-03-29T11:07:16.391 INFO:teuthology.orchestra.run.smithi070.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 6.9 MB/s | 1.1 MB 00:00 2024-03-29T11:07:16.408 INFO:teuthology.orchestra.run.smithi070.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 952 kB/s | 103 kB 00:00 2024-03-29T11:07:16.433 INFO:teuthology.orchestra.run.smithi070.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-29T11:07:16.458 INFO:teuthology.orchestra.run.smithi070.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-03-29T11:07:16.482 INFO:teuthology.orchestra.run.smithi018.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 1.5 MB/s | 247 kB 00:00 2024-03-29T11:07:16.525 INFO:teuthology.orchestra.run.smithi070.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 1.3 MB/s | 125 kB 00:00 2024-03-29T11:07:16.616 INFO:teuthology.orchestra.run.smithi018.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-29T11:07:16.666 INFO:teuthology.orchestra.run.smithi018.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-29T11:07:16.716 INFO:teuthology.orchestra.run.smithi018.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 631 kB/s | 32 kB 00:00 2024-03-29T11:07:16.725 INFO:teuthology.orchestra.run.smithi070.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 755 kB/s | 151 kB 00:00 2024-03-29T11:07:16.750 INFO:teuthology.orchestra.run.smithi070.stdout:(18/36): lua-devel-5.4.4-4.el9.x86_64.rpm 76 kB/s | 22 kB 00:00 2024-03-29T11:07:16.766 INFO:teuthology.orchestra.run.smithi018.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 950 kB/s | 47 kB 00:00 2024-03-29T11:07:16.784 INFO:teuthology.orchestra.run.smithi070.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 740 kB/s | 43 kB 00:00 2024-03-29T11:07:16.808 INFO:teuthology.orchestra.run.smithi018.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 925 kB/s | 39 kB 00:00 2024-03-29T11:07:16.858 INFO:teuthology.orchestra.run.smithi018.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-29T11:07:16.885 INFO:teuthology.orchestra.run.smithi070.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.5 MB/s | 358 kB 00:00 2024-03-29T11:07:16.935 INFO:teuthology.orchestra.run.smithi070.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 214 kB/s | 11 kB 00:00 2024-03-29T11:07:16.969 INFO:teuthology.orchestra.run.smithi070.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 789 kB/s | 172 kB 00:00 2024-03-29T11:07:17.010 INFO:teuthology.orchestra.run.smithi070.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 234 kB/s | 18 kB 00:00 2024-03-29T11:07:17.018 INFO:teuthology.orchestra.run.smithi018.stdout:(11/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 13 MB/s | 17 MB 00:01 2024-03-29T11:07:17.036 INFO:teuthology.orchestra.run.smithi070.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 346 kB/s | 23 kB 00:00 2024-03-29T11:07:17.061 INFO:teuthology.orchestra.run.smithi070.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-03-29T11:07:17.075 INFO:teuthology.orchestra.run.smithi018.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 5.3 MB/s | 1.1 MB 00:00 2024-03-29T11:07:17.095 INFO:teuthology.orchestra.run.smithi070.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 336 kB/s | 20 kB 00:00 2024-03-29T11:07:17.101 INFO:teuthology.orchestra.run.smithi018.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-29T11:07:17.122 INFO:teuthology.orchestra.run.smithi070.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 762 kB/s | 46 kB 00:00 2024-03-29T11:07:17.126 INFO:teuthology.orchestra.run.smithi018.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-29T11:07:17.152 INFO:teuthology.orchestra.run.smithi018.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-29T11:07:17.221 INFO:teuthology.orchestra.run.smithi018.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 1.3 MB/s | 125 kB 00:00 2024-03-29T11:07:17.235 INFO:teuthology.orchestra.run.smithi018.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 267 kB/s | 22 kB 00:00 2024-03-29T11:07:17.289 INFO:teuthology.orchestra.run.smithi070.stdout:(28/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 11 MB/s | 17 MB 00:01 2024-03-29T11:07:17.306 INFO:teuthology.orchestra.run.smithi070.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 373 kB/s | 79 kB 00:00 2024-03-29T11:07:17.331 INFO:teuthology.orchestra.run.smithi070.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.3 MB/s | 272 kB 00:00 2024-03-29T11:07:17.382 INFO:teuthology.orchestra.run.smithi070.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 713 kB/s | 36 kB 00:00 2024-03-29T11:07:17.406 INFO:teuthology.orchestra.run.smithi070.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 893 kB/s | 90 kB 00:00 2024-03-29T11:07:17.426 INFO:teuthology.orchestra.run.smithi070.stdout:(33/36): python3-portend-3.1.0-2.el9.noarch.rpm 122 kB/s | 16 kB 00:00 2024-03-29T11:07:17.449 INFO:teuthology.orchestra.run.smithi070.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.3 MB/s | 230 kB 00:00 2024-03-29T11:07:17.482 INFO:teuthology.orchestra.run.smithi070.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 340 kB/s | 20 kB 00:00 2024-03-29T11:07:17.541 INFO:teuthology.orchestra.run.smithi070.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.1 MB/s | 427 kB 00:00 2024-03-29T11:07:17.544 INFO:teuthology.orchestra.run.smithi018.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 141 kB/s | 43 kB 00:00 2024-03-29T11:07:17.544 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:17.544 INFO:teuthology.orchestra.run.smithi070.stdout:Total 10 MB/s | 30 MB 00:02 2024-03-29T11:07:17.566 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-03-29T11:07:17.605 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-03-29T11:07:17.605 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-03-29T11:07:17.703 INFO:teuthology.orchestra.run.smithi018.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 312 kB/s | 151 kB 00:00 2024-03-29T11:07:17.828 INFO:teuthology.orchestra.run.smithi018.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 606 kB/s | 172 kB 00:00 2024-03-29T11:07:17.936 INFO:teuthology.orchestra.run.smithi018.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2024-03-29T11:07:17.953 INFO:teuthology.orchestra.run.smithi018.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 85 kB/s | 11 kB 00:00 2024-03-29T11:07:18.021 INFO:teuthology.orchestra.run.smithi018.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 211 kB/s | 18 kB 00:00 2024-03-29T11:07:18.038 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-03-29T11:07:18.038 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-03-29T11:07:18.046 INFO:teuthology.orchestra.run.smithi018.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 252 kB/s | 23 kB 00:00 2024-03-29T11:07:18.113 INFO:teuthology.orchestra.run.smithi018.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 212 kB/s | 19 kB 00:00 2024-03-29T11:07:18.138 INFO:teuthology.orchestra.run.smithi018.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 214 kB/s | 20 kB 00:00 2024-03-29T11:07:18.222 INFO:teuthology.orchestra.run.smithi018.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 426 kB/s | 46 kB 00:00 2024-03-29T11:07:18.264 INFO:teuthology.orchestra.run.smithi018.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 629 kB/s | 79 kB 00:00 2024-03-29T11:07:18.356 INFO:teuthology.orchestra.run.smithi018.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-03-29T11:07:18.381 INFO:teuthology.orchestra.run.smithi018.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 140 kB/s | 16 kB 00:00 2024-03-29T11:07:18.464 INFO:teuthology.orchestra.run.smithi018.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 829 kB/s | 90 kB 00:00 2024-03-29T11:07:18.506 INFO:teuthology.orchestra.run.smithi018.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 285 kB/s | 36 kB 00:00 2024-03-29T11:07:18.607 INFO:teuthology.orchestra.run.smithi018.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.6 MB/s | 230 kB 00:00 2024-03-29T11:07:18.707 INFO:teuthology.orchestra.run.smithi018.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 200 kB/s | 20 kB 00:00 2024-03-29T11:07:18.732 INFO:teuthology.orchestra.run.smithi018.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.8 MB/s | 427 kB 00:00 2024-03-29T11:07:19.024 INFO:teuthology.orchestra.run.smithi018.stdout:(36/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 1.3 MB/s | 4.7 MB 00:03 2024-03-29T11:07:19.028 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:19.028 INFO:teuthology.orchestra.run.smithi018.stdout:Total 7.1 MB/s | 30 MB 00:04 2024-03-29T11:07:19.053 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-29T11:07:19.100 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-29T11:07:19.100 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-29T11:07:19.341 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-03-29T11:07:19.402 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:07:19.446 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:07:19.471 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:07:19.532 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:07:19.552 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-29T11:07:19.552 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-29T11:07:19.569 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:07:19.602 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:07:19.693 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:07:19.863 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:07:19.913 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:07:19.984 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:07:20.039 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:07:20.078 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:07:20.126 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:07:20.189 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:07:20.372 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:07:20.415 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:07:20.445 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:07:20.480 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:07:20.519 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:07:20.595 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:07:20.638 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:07:20.671 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:07:20.711 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:07:20.746 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:07:20.789 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:07:20.877 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:07:20.902 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-29T11:07:20.920 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:07:20.968 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:07:20.991 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:07:21.002 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:07:21.029 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:07:21.102 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:07:21.128 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:07:21.153 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:07:21.186 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:07:21.253 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:07:21.282 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:07:21.367 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:07:21.412 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:07:21.414 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:07:21.442 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:07:21.468 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:07:21.468 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:07:21.468 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:07:21.468 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:07:21.468 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:07:21.468 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:07:21.488 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:07:21.541 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:07:21.578 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:07:21.636 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:07:21.691 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:07:21.884 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:07:21.932 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:07:21.971 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:07:22.014 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:07:22.044 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:07:22.123 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:07:22.154 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:07:22.187 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:07:22.237 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:07:22.272 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:07:22.307 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:07:22.398 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:07:22.437 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:07:22.513 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:07:22.662 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:07:22.786 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:07:22.893 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:07:22.919 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:07:22.919 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:07:22.919 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:07:22.919 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:07:22.919 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:07:22.919 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:07:22.961 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:07:22.992 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:07:23.016 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:07:23.017 INFO:teuthology.orchestra.run.smithi018.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:07:23.017 INFO:teuthology.orchestra.run.smithi018.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:07:23.017 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:07:23.017 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:07:23.017 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:07:23.360 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:07:23.384 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:07:23.384 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:07:23.385 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:07:23.385 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:07:23.385 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:07:23.385 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:07:23.594 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:07:23.617 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:07:23.617 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:07:23.617 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:07:23.617 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:07:23.617 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:07:23.617 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:07:23.684 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:07:24.526 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:07:24.552 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:07:24.552 INFO:teuthology.orchestra.run.smithi018.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:07:24.552 INFO:teuthology.orchestra.run.smithi018.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:07:24.552 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:07:24.553 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:07:24.553 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:07:25.018 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:07:25.041 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:07:25.041 INFO:teuthology.orchestra.run.smithi018.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:07:25.041 INFO:teuthology.orchestra.run.smithi018.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:07:25.041 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:07:25.041 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:07:25.041 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:07:25.269 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:07:25.292 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:07:25.292 INFO:teuthology.orchestra.run.smithi018.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:07:25.292 INFO:teuthology.orchestra.run.smithi018.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:07:25.292 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:07:25.292 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:07:25.292 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:07:25.360 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:07:25.391 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:07:25.391 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:07:25.391 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:07:25.391 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:07:25.391 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:07:25.391 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:07:25.391 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:07:25.391 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:07:25.391 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:07:25.391 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:07:25.391 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:07:25.392 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:07:25.392 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:07:25.392 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:07:25.392 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:07:25.392 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:07:25.392 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:07:25.392 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:07:25.392 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:07:25.392 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:07:25.392 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:07:25.393 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:07:25.393 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:07:25.393 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:07:25.393 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:07:25.393 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:07:25.393 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:07:25.393 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:07:25.393 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:07:25.394 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:07:25.394 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:07:25.394 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:07:25.394 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:07:25.394 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:07:25.394 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:07:25.395 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:07:26.454 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:07:26.454 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:07:26.454 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-03-29T11:07:26.454 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:07:26.454 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:07:26.454 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:07:26.454 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:07:26.455 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:07:26.455 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:07:26.455 INFO:teuthology.orchestra.run.smithi070.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:07:26.455 INFO:teuthology.orchestra.run.smithi070.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:07:26.455 INFO:teuthology.orchestra.run.smithi070.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:07:26.455 INFO:teuthology.orchestra.run.smithi070.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:07:26.455 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:07:26.455 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:07:26.455 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:07:26.455 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:07:26.455 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:07:26.455 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:07:26.455 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:07:26.455 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:07:26.456 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:07:26.456 INFO:teuthology.orchestra.run.smithi070.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:07:26.456 INFO:teuthology.orchestra.run.smithi070.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:07:26.456 INFO:teuthology.orchestra.run.smithi070.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:07:26.456 INFO:teuthology.orchestra.run.smithi070.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:07:26.456 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:07:26.456 INFO:teuthology.orchestra.run.smithi070.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:07:26.456 INFO:teuthology.orchestra.run.smithi070.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:07:26.456 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:07:26.456 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:07:26.456 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:07:26.456 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:07:26.456 INFO:teuthology.orchestra.run.smithi070.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:07:26.456 INFO:teuthology.orchestra.run.smithi070.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:07:26.456 INFO:teuthology.orchestra.run.smithi070.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:07:26.457 INFO:teuthology.orchestra.run.smithi070.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:07:26.457 INFO:teuthology.orchestra.run.smithi070.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:07:26.457 INFO:teuthology.orchestra.run.smithi070.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:07:26.457 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:07:26.457 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-03-29T11:07:26.812 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-base 2024-03-29T11:07:27.063 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:07:27.063 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:07:27.063 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:07:27.063 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:07:27.063 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:07:27.064 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:07:27.064 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:07:27.064 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:07:27.064 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:07:27.064 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:07:27.064 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:07:27.064 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:07:27.064 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:07:27.064 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:07:27.064 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:07:27.064 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:07:27.064 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:07:27.064 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:07:27.064 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:07:27.065 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:07:27.065 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:07:27.065 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:07:27.065 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:07:27.065 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:07:27.065 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:07:27.065 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:07:27.065 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:07:27.065 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:07:27.065 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:07:27.065 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:07:27.065 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:07:27.065 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:07:27.065 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:07:27.065 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:07:27.066 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:07:27.066 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:07:27.379 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:00:54 ago on Fri 29 Mar 2024 11:06:33 AM UTC. 2024-03-29T11:07:27.478 INFO:teuthology.orchestra.run.smithi070.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:07:27.531 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-03-29T11:07:27.532 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-03-29T11:07:27.532 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-03-29T11:07:27.602 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install cephadm 2024-03-29T11:07:28.143 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:07:28.144 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:07:28.144 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-29T11:07:28.144 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:07:28.144 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:07:28.144 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:07:28.144 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:07:28.144 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:07:28.144 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:07:28.144 INFO:teuthology.orchestra.run.smithi018.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:07:28.144 INFO:teuthology.orchestra.run.smithi018.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:07:28.144 INFO:teuthology.orchestra.run.smithi018.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:07:28.144 INFO:teuthology.orchestra.run.smithi018.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:07:28.145 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:07:28.145 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:07:28.145 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:07:28.145 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:07:28.145 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:07:28.145 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:07:28.145 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:07:28.145 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:07:28.145 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:07:28.145 INFO:teuthology.orchestra.run.smithi018.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:07:28.145 INFO:teuthology.orchestra.run.smithi018.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:07:28.145 INFO:teuthology.orchestra.run.smithi018.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:07:28.145 INFO:teuthology.orchestra.run.smithi018.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:07:28.145 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:07:28.146 INFO:teuthology.orchestra.run.smithi018.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:07:28.146 INFO:teuthology.orchestra.run.smithi018.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:07:28.146 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:07:28.146 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:07:28.146 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:07:28.146 INFO:teuthology.orchestra.run.smithi018.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:07:28.146 INFO:teuthology.orchestra.run.smithi018.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:07:28.146 INFO:teuthology.orchestra.run.smithi018.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:07:28.146 INFO:teuthology.orchestra.run.smithi018.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:07:28.146 INFO:teuthology.orchestra.run.smithi018.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:07:28.146 INFO:teuthology.orchestra.run.smithi018.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:07:28.146 INFO:teuthology.orchestra.run.smithi018.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:07:28.146 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:07:28.146 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-29T11:07:28.167 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:00:55 ago on Fri 29 Mar 2024 11:06:33 AM UTC. 2024-03-29T11:07:28.318 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-03-29T11:07:28.319 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-03-29T11:07:28.319 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-03-29T11:07:28.319 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-03-29T11:07:28.319 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-03-29T11:07:28.319 INFO:teuthology.orchestra.run.smithi070.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:07:28.319 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:07:28.319 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-03-29T11:07:28.319 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-03-29T11:07:28.319 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2024-03-29T11:07:28.319 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:07:28.320 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 747 k 2024-03-29T11:07:28.320 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 754 k 2024-03-29T11:07:28.320 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-03-29T11:07:28.479 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-base 2024-03-29T11:07:28.862 INFO:teuthology.orchestra.run.smithi070.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.3 MB/s | 747 kB 00:00 2024-03-29T11:07:28.863 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:28.863 INFO:teuthology.orchestra.run.smithi070.stdout:Total 1.3 MB/s | 747 kB 00:00 2024-03-29T11:07:28.863 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-03-29T11:07:28.869 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-03-29T11:07:28.869 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-03-29T11:07:28.874 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-03-29T11:07:28.875 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-03-29T11:07:28.951 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-03-29T11:07:29.057 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:00:58 ago on Fri 29 Mar 2024 11:06:31 AM UTC. 2024-03-29T11:07:29.159 INFO:teuthology.orchestra.run.smithi018.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:07:29.214 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-29T11:07:29.216 INFO:teuthology.orchestra.run.smithi018.stdout:Nothing to do. 2024-03-29T11:07:29.216 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-29T11:07:29.284 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install cephadm 2024-03-29T11:07:29.379 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:07:29.467 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:07:29.858 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:00:58 ago on Fri 29 Mar 2024 11:06:31 AM UTC. 2024-03-29T11:07:29.859 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:07:30.015 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-29T11:07:30.015 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-29T11:07:30.015 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repository Size 2024-03-29T11:07:30.016 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-29T11:07:30.016 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-29T11:07:30.016 INFO:teuthology.orchestra.run.smithi018.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:07:30.016 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:07:30.016 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-29T11:07:30.016 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-29T11:07:30.016 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-29T11:07:30.016 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:07:30.016 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 747 k 2024-03-29T11:07:30.016 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 754 k 2024-03-29T11:07:30.016 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-29T11:07:30.125 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:07:30.125 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:07:30.125 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-03-29T11:07:30.125 INFO:teuthology.orchestra.run.smithi070.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:07:30.126 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:07:30.126 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-03-29T11:07:30.309 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:07:30.537 INFO:teuthology.orchestra.run.smithi018.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.4 MB/s | 747 kB 00:00 2024-03-29T11:07:30.537 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:30.538 INFO:teuthology.orchestra.run.smithi018.stdout:Total 1.4 MB/s | 747 kB 00:00 2024-03-29T11:07:30.538 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-29T11:07:30.543 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-29T11:07:30.543 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-29T11:07:30.549 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-29T11:07:30.550 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-29T11:07:30.625 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-29T11:07:30.878 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:00:57 ago on Fri 29 Mar 2024 11:06:33 AM UTC. 2024-03-29T11:07:31.032 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-03-29T11:07:31.033 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-03-29T11:07:31.033 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repo Size 2024-03-29T11:07:31.033 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-03-29T11:07:31.033 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-03-29T11:07:31.033 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:07:31.033 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:07:31.033 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-03-29T11:07:31.033 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-03-29T11:07:31.033 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2024-03-29T11:07:31.033 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:07:31.033 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 145 k 2024-03-29T11:07:31.033 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 436 k 2024-03-29T11:07:31.034 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-03-29T11:07:31.046 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:07:31.128 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:07:31.277 INFO:teuthology.orchestra.run.smithi070.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 599 kB/s | 145 kB 00:00 2024-03-29T11:07:31.277 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:31.277 INFO:teuthology.orchestra.run.smithi070.stdout:Total 595 kB/s | 145 kB 00:00 2024-03-29T11:07:31.277 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-03-29T11:07:31.285 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-03-29T11:07:31.285 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-03-29T11:07:31.337 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-03-29T11:07:31.337 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-03-29T11:07:31.471 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-03-29T11:07:31.519 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:07:31.544 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:07:31.544 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:07:31.544 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:07:31.544 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:07:31.604 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:07:31.886 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:07:31.886 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:07:31.886 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-29T11:07:31.886 INFO:teuthology.orchestra.run.smithi018.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:07:31.886 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:07:31.887 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-29T11:07:32.066 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:07:32.508 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:07:32.508 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:07:32.508 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-03-29T11:07:32.508 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:07:32.508 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:07:32.508 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-03-29T11:07:32.639 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:01:01 ago on Fri 29 Mar 2024 11:06:31 AM UTC. 2024-03-29T11:07:32.740 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-mgr 2024-03-29T11:07:32.797 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-29T11:07:32.798 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-29T11:07:32.798 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-29T11:07:32.798 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-29T11:07:32.798 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-29T11:07:32.798 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:07:32.798 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:07:32.798 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-29T11:07:32.798 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-29T11:07:32.798 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-29T11:07:32.799 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:07:32.799 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 145 k 2024-03-29T11:07:32.799 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 436 k 2024-03-29T11:07:32.799 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-29T11:07:33.168 INFO:teuthology.orchestra.run.smithi018.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 394 kB/s | 145 kB 00:00 2024-03-29T11:07:33.169 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:33.169 INFO:teuthology.orchestra.run.smithi018.stdout:Total 392 kB/s | 145 kB 00:00 2024-03-29T11:07:33.169 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-29T11:07:33.177 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-29T11:07:33.177 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-29T11:07:33.233 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-29T11:07:33.233 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-29T11:07:33.315 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:00 ago on Fri 29 Mar 2024 11:06:33 AM UTC. 2024-03-29T11:07:33.375 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-29T11:07:33.415 INFO:teuthology.orchestra.run.smithi070.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:07:33.419 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:07:33.443 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:07:33.444 INFO:teuthology.orchestra.run.smithi018.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:07:33.444 INFO:teuthology.orchestra.run.smithi018.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:07:33.444 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:07:33.468 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-03-29T11:07:33.468 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-03-29T11:07:33.469 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-03-29T11:07:33.536 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:07:34.092 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:01 ago on Fri 29 Mar 2024 11:06:33 AM UTC. 2024-03-29T11:07:34.243 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-03-29T11:07:34.244 INFO:teuthology.orchestra.run.smithi070.stdout:====================================================================================== 2024-03-29T11:07:34.244 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-03-29T11:07:34.244 INFO:teuthology.orchestra.run.smithi070.stdout:====================================================================================== 2024-03-29T11:07:34.244 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-03-29T11:07:34.244 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:07:34.244 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-03-29T11:07:34.244 INFO:teuthology.orchestra.run.smithi070.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:07:34.244 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:07:34.244 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:07:34.244 INFO:teuthology.orchestra.run.smithi070.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:07:34.245 INFO:teuthology.orchestra.run.smithi070.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:07:34.245 INFO:teuthology.orchestra.run.smithi070.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:07:34.245 INFO:teuthology.orchestra.run.smithi070.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:07:34.245 INFO:teuthology.orchestra.run.smithi070.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:07:34.245 INFO:teuthology.orchestra.run.smithi070.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:07:34.245 INFO:teuthology.orchestra.run.smithi070.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:07:34.245 INFO:teuthology.orchestra.run.smithi070.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:07:34.245 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:07:34.245 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-03-29T11:07:34.245 INFO:teuthology.orchestra.run.smithi070.stdout:====================================================================================== 2024-03-29T11:07:34.245 INFO:teuthology.orchestra.run.smithi070.stdout:Install 12 Packages 2024-03-29T11:07:34.245 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:07:34.246 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 6.9 M 2024-03-29T11:07:34.246 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 26 M 2024-03-29T11:07:34.246 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-03-29T11:07:34.461 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:07:34.461 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:07:34.461 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-29T11:07:34.461 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:07:34.461 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:07:34.462 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-29T11:07:34.731 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-mgr 2024-03-29T11:07:34.830 INFO:teuthology.orchestra.run.smithi070.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 158 kB/s | 31 kB 00:00 2024-03-29T11:07:34.939 INFO:teuthology.orchestra.run.smithi070.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 50 kB/s | 15 kB 00:00 2024-03-29T11:07:35.006 INFO:teuthology.orchestra.run.smithi070.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.8 MB/s | 1.0 MB 00:00 2024-03-29T11:07:35.031 INFO:teuthology.orchestra.run.smithi070.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.5 MB/s | 238 kB 00:00 2024-03-29T11:07:35.290 INFO:teuthology.orchestra.run.smithi070.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.1 MB/s | 548 kB 00:00 2024-03-29T11:07:35.309 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:01:04 ago on Fri 29 Mar 2024 11:06:31 AM UTC. 2024-03-29T11:07:35.323 INFO:teuthology.orchestra.run.smithi070.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.7 MB/s | 863 kB 00:00 2024-03-29T11:07:35.349 INFO:teuthology.orchestra.run.smithi070.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 332 kB/s | 19 kB 00:00 2024-03-29T11:07:35.411 INFO:teuthology.orchestra.run.smithi018.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:07:35.441 INFO:teuthology.orchestra.run.smithi070.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 17 MB/s | 2.0 MB 00:00 2024-03-29T11:07:35.458 INFO:teuthology.orchestra.run.smithi070.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.3 MB/s | 144 kB 00:00 2024-03-29T11:07:35.467 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-29T11:07:35.468 INFO:teuthology.orchestra.run.smithi018.stdout:Nothing to do. 2024-03-29T11:07:35.468 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-29T11:07:35.491 INFO:teuthology.orchestra.run.smithi070.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 613 kB/s | 31 kB 00:00 2024-03-29T11:07:35.516 INFO:teuthology.orchestra.run.smithi070.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.1 MB/s | 188 kB 00:00 2024-03-29T11:07:35.547 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:07:35.558 INFO:teuthology.orchestra.run.smithi070.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 1.9 MB/s | 1.8 MB 00:00 2024-03-29T11:07:35.563 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:35.563 INFO:teuthology.orchestra.run.smithi070.stdout:Total 5.2 MB/s | 6.9 MB 00:01 2024-03-29T11:07:35.581 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-03-29T11:07:35.597 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-03-29T11:07:35.597 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-03-29T11:07:35.773 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-03-29T11:07:35.773 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-03-29T11:07:36.112 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:01:05 ago on Fri 29 Mar 2024 11:06:31 AM UTC. 2024-03-29T11:07:36.269 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-29T11:07:36.270 INFO:teuthology.orchestra.run.smithi018.stdout:====================================================================================== 2024-03-29T11:07:36.270 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repository Size 2024-03-29T11:07:36.270 INFO:teuthology.orchestra.run.smithi018.stdout:====================================================================================== 2024-03-29T11:07:36.271 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-29T11:07:36.271 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:07:36.271 INFO:teuthology.orchestra.run.smithi018.stdout:Installing dependencies: 2024-03-29T11:07:36.271 INFO:teuthology.orchestra.run.smithi018.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:07:36.271 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:07:36.271 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:07:36.271 INFO:teuthology.orchestra.run.smithi018.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:07:36.271 INFO:teuthology.orchestra.run.smithi018.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:07:36.271 INFO:teuthology.orchestra.run.smithi018.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:07:36.272 INFO:teuthology.orchestra.run.smithi018.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:07:36.272 INFO:teuthology.orchestra.run.smithi018.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:07:36.272 INFO:teuthology.orchestra.run.smithi018.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:07:36.272 INFO:teuthology.orchestra.run.smithi018.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:07:36.272 INFO:teuthology.orchestra.run.smithi018.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:07:36.272 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:07:36.272 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-29T11:07:36.272 INFO:teuthology.orchestra.run.smithi018.stdout:====================================================================================== 2024-03-29T11:07:36.272 INFO:teuthology.orchestra.run.smithi018.stdout:Install 12 Packages 2024-03-29T11:07:36.272 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:07:36.273 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 6.9 M 2024-03-29T11:07:36.273 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 26 M 2024-03-29T11:07:36.274 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-29T11:07:36.279 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-03-29T11:07:36.351 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:07:36.394 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:07:36.440 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:07:36.479 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:07:36.586 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:07:36.605 INFO:teuthology.orchestra.run.smithi018.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 164 kB/s | 31 kB 00:00 2024-03-29T11:07:36.704 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:07:36.706 INFO:teuthology.orchestra.run.smithi018.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 52 kB/s | 15 kB 00:00 2024-03-29T11:07:36.765 INFO:teuthology.orchestra.run.smithi018.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.4 MB/s | 1.0 MB 00:00 2024-03-29T11:07:36.790 INFO:teuthology.orchestra.run.smithi018.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.8 MB/s | 238 kB 00:00 2024-03-29T11:07:36.797 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:07:36.851 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:07:36.879 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:07:36.911 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:07:37.015 INFO:teuthology.orchestra.run.smithi018.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.4 MB/s | 863 kB 00:00 2024-03-29T11:07:37.057 INFO:teuthology.orchestra.run.smithi018.stdout:(6/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 2.8 MB/s | 1.8 MB 00:00 2024-03-29T11:07:37.153 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:07:37.195 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:07:37.391 INFO:teuthology.orchestra.run.smithi018.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 52 kB/s | 19 kB 00:00 2024-03-29T11:07:37.532 INFO:teuthology.orchestra.run.smithi018.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 738 kB/s | 548 kB 00:00 2024-03-29T11:07:37.633 INFO:teuthology.orchestra.run.smithi018.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 308 kB/s | 31 kB 00:00 2024-03-29T11:07:37.708 INFO:teuthology.orchestra.run.smithi018.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 455 kB/s | 144 kB 00:00 2024-03-29T11:07:37.758 INFO:teuthology.orchestra.run.smithi018.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.5 MB/s | 188 kB 00:00 2024-03-29T11:07:37.984 INFO:teuthology.orchestra.run.smithi018.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.2 MB/s | 2.0 MB 00:00 2024-03-29T11:07:37.989 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:37.990 INFO:teuthology.orchestra.run.smithi018.stdout:Total 4.0 MB/s | 6.9 MB 00:01 2024-03-29T11:07:38.009 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-29T11:07:38.030 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-29T11:07:38.030 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-29T11:07:38.103 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:07:38.103 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:07:38.103 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:07:38.103 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:07:38.103 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:07:38.103 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:07:38.103 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:07:38.103 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:07:38.103 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:07:38.103 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:07:38.103 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:07:38.104 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:07:38.221 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-29T11:07:38.221 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-29T11:07:38.471 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:07:38.472 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:07:38.472 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-03-29T11:07:38.472 INFO:teuthology.orchestra.run.smithi070.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:07:38.472 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:07:38.472 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:07:38.472 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:07:38.472 INFO:teuthology.orchestra.run.smithi070.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:07:38.472 INFO:teuthology.orchestra.run.smithi070.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:07:38.472 INFO:teuthology.orchestra.run.smithi070.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:07:38.472 INFO:teuthology.orchestra.run.smithi070.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:07:38.472 INFO:teuthology.orchestra.run.smithi070.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:07:38.472 INFO:teuthology.orchestra.run.smithi070.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:07:38.472 INFO:teuthology.orchestra.run.smithi070.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:07:38.472 INFO:teuthology.orchestra.run.smithi070.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:07:38.473 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:07:38.473 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-03-29T11:07:38.734 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:07:38.863 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-29T11:07:38.934 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:07:38.969 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:07:39.016 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:07:39.054 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:07:39.171 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:07:39.292 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:06 ago on Fri 29 Mar 2024 11:06:33 AM UTC. 2024-03-29T11:07:39.300 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:07:39.427 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:07:39.445 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-03-29T11:07:39.446 INFO:teuthology.orchestra.run.smithi070.stdout:============================================================================================ 2024-03-29T11:07:39.446 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-03-29T11:07:39.446 INFO:teuthology.orchestra.run.smithi070.stdout:============================================================================================ 2024-03-29T11:07:39.446 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-03-29T11:07:39.447 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:07:39.447 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-03-29T11:07:39.447 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:07:39.447 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:07:39.447 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:07:39.447 INFO:teuthology.orchestra.run.smithi070.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:07:39.447 INFO:teuthology.orchestra.run.smithi070.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:07:39.447 INFO:teuthology.orchestra.run.smithi070.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:07:39.447 INFO:teuthology.orchestra.run.smithi070.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:07:39.447 INFO:teuthology.orchestra.run.smithi070.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:07:39.447 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:07:39.447 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:07:39.447 INFO:teuthology.orchestra.run.smithi070.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:07:39.447 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:07:39.448 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-03-29T11:07:39.448 INFO:teuthology.orchestra.run.smithi070.stdout:============================================================================================ 2024-03-29T11:07:39.448 INFO:teuthology.orchestra.run.smithi070.stdout:Install 12 Packages 2024-03-29T11:07:39.448 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:07:39.448 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 37 M 2024-03-29T11:07:39.448 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 209 M 2024-03-29T11:07:39.448 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-03-29T11:07:39.585 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:07:39.612 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:07:39.645 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:07:39.832 INFO:teuthology.orchestra.run.smithi070.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 232 kB/s | 32 kB 00:00 2024-03-29T11:07:39.866 INFO:teuthology.orchestra.run.smithi070.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 535 kB/s | 17 kB 00:00 2024-03-29T11:07:39.898 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:07:39.908 INFO:teuthology.orchestra.run.smithi070.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 846 kB/s | 34 kB 00:00 2024-03-29T11:07:39.927 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:07:39.958 INFO:teuthology.orchestra.run.smithi070.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-29T11:07:40.158 INFO:teuthology.orchestra.run.smithi070.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 19 MB/s | 4.7 MB 00:00 2024-03-29T11:07:40.233 INFO:teuthology.orchestra.run.smithi070.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 4.8 MB 00:00 2024-03-29T11:07:40.258 INFO:teuthology.orchestra.run.smithi070.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.1 MB/s | 217 kB 00:00 2024-03-29T11:07:40.592 INFO:teuthology.orchestra.run.smithi070.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 44 MB/s | 16 MB 00:00 2024-03-29T11:07:40.676 INFO:teuthology.orchestra.run.smithi070.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 195 kB 00:00 2024-03-29T11:07:40.701 INFO:teuthology.orchestra.run.smithi070.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.8 MB/s | 803 kB 00:00 2024-03-29T11:07:40.703 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:07:40.703 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:07:40.703 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:07:40.704 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:07:40.704 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:07:40.704 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:07:40.704 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:07:40.704 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:07:40.704 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:07:40.704 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:07:40.704 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:07:40.704 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:07:40.818 INFO:teuthology.orchestra.run.smithi070.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.7 MB/s | 245 kB 00:00 2024-03-29T11:07:41.044 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:07:41.044 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:07:41.044 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-29T11:07:41.044 INFO:teuthology.orchestra.run.smithi018.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:07:41.044 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:07:41.044 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:07:41.044 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:07:41.044 INFO:teuthology.orchestra.run.smithi018.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:07:41.045 INFO:teuthology.orchestra.run.smithi018.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:07:41.045 INFO:teuthology.orchestra.run.smithi018.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:07:41.045 INFO:teuthology.orchestra.run.smithi018.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:07:41.045 INFO:teuthology.orchestra.run.smithi018.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:07:41.045 INFO:teuthology.orchestra.run.smithi018.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:07:41.045 INFO:teuthology.orchestra.run.smithi018.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:07:41.045 INFO:teuthology.orchestra.run.smithi018.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:07:41.045 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:07:41.045 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-29T11:07:41.271 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:07:41.418 INFO:teuthology.orchestra.run.smithi070.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 4.3 MB/s | 7.4 MB 00:01 2024-03-29T11:07:41.421 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:41.421 INFO:teuthology.orchestra.run.smithi070.stdout:Total 19 MB/s | 37 MB 00:01 2024-03-29T11:07:41.462 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-03-29T11:07:41.486 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-03-29T11:07:41.486 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-03-29T11:07:41.846 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-03-29T11:07:41.846 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-03-29T11:07:41.848 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:01:10 ago on Fri 29 Mar 2024 11:06:31 AM UTC. 2024-03-29T11:07:42.007 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-29T11:07:42.008 INFO:teuthology.orchestra.run.smithi018.stdout:============================================================================================ 2024-03-29T11:07:42.008 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repository Size 2024-03-29T11:07:42.008 INFO:teuthology.orchestra.run.smithi018.stdout:============================================================================================ 2024-03-29T11:07:42.008 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-29T11:07:42.008 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:07:42.008 INFO:teuthology.orchestra.run.smithi018.stdout:Installing dependencies: 2024-03-29T11:07:42.008 INFO:teuthology.orchestra.run.smithi018.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:07:42.008 INFO:teuthology.orchestra.run.smithi018.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:07:42.008 INFO:teuthology.orchestra.run.smithi018.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:07:42.008 INFO:teuthology.orchestra.run.smithi018.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:07:42.009 INFO:teuthology.orchestra.run.smithi018.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:07:42.009 INFO:teuthology.orchestra.run.smithi018.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:07:42.009 INFO:teuthology.orchestra.run.smithi018.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:07:42.009 INFO:teuthology.orchestra.run.smithi018.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:07:42.009 INFO:teuthology.orchestra.run.smithi018.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:07:42.009 INFO:teuthology.orchestra.run.smithi018.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:07:42.009 INFO:teuthology.orchestra.run.smithi018.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:07:42.009 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:07:42.009 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-29T11:07:42.009 INFO:teuthology.orchestra.run.smithi018.stdout:============================================================================================ 2024-03-29T11:07:42.009 INFO:teuthology.orchestra.run.smithi018.stdout:Install 12 Packages 2024-03-29T11:07:42.009 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:07:42.010 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 37 M 2024-03-29T11:07:42.010 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 209 M 2024-03-29T11:07:42.010 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-29T11:07:42.573 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-03-29T11:07:42.658 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:07:42.680 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:07:42.757 INFO:teuthology.orchestra.run.smithi018.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 154 kB/s | 32 kB 00:00 2024-03-29T11:07:42.773 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:07:43.148 INFO:teuthology.orchestra.run.smithi018.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 293 kB/s | 17 kB 00:00 2024-03-29T11:07:43.149 INFO:teuthology.orchestra.run.smithi018.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 682 kB/s | 34 kB 00:00 2024-03-29T11:07:43.149 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:07:43.149 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:07:43.149 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:07:43.149 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:07:43.358 INFO:teuthology.orchestra.run.smithi018.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 9.6 MB/s | 4.7 MB 00:00 2024-03-29T11:07:43.608 INFO:teuthology.orchestra.run.smithi018.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-03-29T11:07:43.658 INFO:teuthology.orchestra.run.smithi018.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 4.3 MB/s | 217 kB 00:00 2024-03-29T11:07:43.741 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:07:43.825 INFO:teuthology.orchestra.run.smithi018.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.3 MB/s | 3.0 MB 00:01 2024-03-29T11:07:43.861 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:07:44.359 INFO:teuthology.orchestra.run.smithi018.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 22 MB/s | 16 MB 00:00 2024-03-29T11:07:44.401 INFO:teuthology.orchestra.run.smithi018.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 803 kB 00:00 2024-03-29T11:07:44.446 INFO:teuthology.orchestra.run.smithi018.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 195 kB 00:00 2024-03-29T11:07:44.576 INFO:teuthology.orchestra.run.smithi018.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.4 MB/s | 245 kB 00:00 2024-03-29T11:07:44.702 INFO:teuthology.orchestra.run.smithi018.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 3.4 MB/s | 7.4 MB 00:02 2024-03-29T11:07:44.704 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:44.705 INFO:teuthology.orchestra.run.smithi018.stdout:Total 14 MB/s | 37 MB 00:02 2024-03-29T11:07:44.746 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-29T11:07:44.771 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-29T11:07:44.771 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-29T11:07:45.094 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:07:45.148 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-29T11:07:45.149 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-29T11:07:45.983 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-29T11:07:46.075 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:07:46.106 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:07:46.107 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:07:46.141 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:07:46.205 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:07:46.243 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:07:46.429 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:07:46.472 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:07:46.563 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:07:47.263 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:07:47.376 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:07:48.654 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:07:48.654 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:07:48.654 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:07:48.654 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:07:48.654 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:07:48.654 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:07:48.654 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:07:48.654 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:07:48.654 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:07:48.654 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:07:48.654 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:07:48.655 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:07:48.677 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:07:49.082 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:07:49.082 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:07:49.082 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-03-29T11:07:49.082 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:07:49.082 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:07:49.083 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:07:49.083 INFO:teuthology.orchestra.run.smithi070.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:07:49.083 INFO:teuthology.orchestra.run.smithi070.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:07:49.083 INFO:teuthology.orchestra.run.smithi070.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:07:49.083 INFO:teuthology.orchestra.run.smithi070.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:07:49.083 INFO:teuthology.orchestra.run.smithi070.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:07:49.083 INFO:teuthology.orchestra.run.smithi070.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:07:49.083 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:07:49.083 INFO:teuthology.orchestra.run.smithi070.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:07:49.083 INFO:teuthology.orchestra.run.smithi070.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:07:49.083 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:07:49.083 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-03-29T11:07:49.467 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:07:49.744 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:07:49.775 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:07:50.040 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:17 ago on Fri 29 Mar 2024 11:06:33 AM UTC. 2024-03-29T11:07:50.191 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-03-29T11:07:50.192 INFO:teuthology.orchestra.run.smithi070.stdout:======================================================================================== 2024-03-29T11:07:50.192 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-03-29T11:07:50.192 INFO:teuthology.orchestra.run.smithi070.stdout:======================================================================================== 2024-03-29T11:07:50.192 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-03-29T11:07:50.192 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:07:50.193 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-03-29T11:07:50.193 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:07:50.193 INFO:teuthology.orchestra.run.smithi070.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:07:50.193 INFO:teuthology.orchestra.run.smithi070.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:07:50.193 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:07:50.193 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:07:50.193 INFO:teuthology.orchestra.run.smithi070.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:07:50.193 INFO:teuthology.orchestra.run.smithi070.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:07:50.193 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:07:50.193 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:07:50.193 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:07:50.194 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:07:50.194 INFO:teuthology.orchestra.run.smithi070.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:07:50.194 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:07:50.194 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-03-29T11:07:50.194 INFO:teuthology.orchestra.run.smithi070.stdout:======================================================================================== 2024-03-29T11:07:50.194 INFO:teuthology.orchestra.run.smithi070.stdout:Install 13 Packages 2024-03-29T11:07:50.194 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:07:50.195 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 2.0 M 2024-03-29T11:07:50.195 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 26 M 2024-03-29T11:07:50.195 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-03-29T11:07:50.401 INFO:teuthology.orchestra.run.smithi070.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 188 kB/s | 24 kB 00:00 2024-03-29T11:07:50.426 INFO:teuthology.orchestra.run.smithi070.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 116 kB/s | 18 kB 00:00 2024-03-29T11:07:50.468 INFO:teuthology.orchestra.run.smithi070.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-29T11:07:50.493 INFO:teuthology.orchestra.run.smithi070.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-29T11:07:50.519 INFO:teuthology.orchestra.run.smithi070.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:07:50.544 INFO:teuthology.orchestra.run.smithi070.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2024-03-29T11:07:50.569 INFO:teuthology.orchestra.run.smithi070.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 175 kB/s | 52 kB 00:00 2024-03-29T11:07:50.661 INFO:teuthology.orchestra.run.smithi070.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 128 kB/s | 15 kB 00:00 2024-03-29T11:07:50.686 INFO:teuthology.orchestra.run.smithi070.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 192 kB/s | 32 kB 00:00 2024-03-29T11:07:50.803 INFO:teuthology.orchestra.run.smithi070.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 866 kB/s | 203 kB 00:00 2024-03-29T11:07:50.828 INFO:teuthology.orchestra.run.smithi070.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 416 kB/s | 59 kB 00:00 2024-03-29T11:07:50.887 INFO:teuthology.orchestra.run.smithi070.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.0 MB/s | 90 kB 00:00 2024-03-29T11:07:50.954 INFO:teuthology.orchestra.run.smithi070.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.5 MB/s | 1.0 MB 00:00 2024-03-29T11:07:50.956 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:50.956 INFO:teuthology.orchestra.run.smithi070.stdout:Total 2.7 MB/s | 2.0 MB 00:00 2024-03-29T11:07:50.956 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-03-29T11:07:50.969 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-03-29T11:07:50.969 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-03-29T11:07:51.060 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-03-29T11:07:51.060 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-03-29T11:07:51.574 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-03-29T11:07:51.750 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:07:51.806 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:07:51.860 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:07:51.898 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:07:51.939 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:07:52.032 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:07:52.134 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:07:52.182 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:07:52.407 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:07:52.407 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:07:52.407 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:07:52.407 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:07:52.407 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:07:52.407 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:07:52.407 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:07:52.407 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:07:52.407 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:07:52.408 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:07:52.408 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:07:52.408 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:07:52.838 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:07:52.838 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:07:52.838 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-29T11:07:52.838 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:07:52.838 INFO:teuthology.orchestra.run.smithi018.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:07:52.839 INFO:teuthology.orchestra.run.smithi018.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:07:52.839 INFO:teuthology.orchestra.run.smithi018.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:07:52.839 INFO:teuthology.orchestra.run.smithi018.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:07:52.839 INFO:teuthology.orchestra.run.smithi018.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:07:52.839 INFO:teuthology.orchestra.run.smithi018.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:07:52.839 INFO:teuthology.orchestra.run.smithi018.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:07:52.839 INFO:teuthology.orchestra.run.smithi018.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:07:52.839 INFO:teuthology.orchestra.run.smithi018.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:07:52.839 INFO:teuthology.orchestra.run.smithi018.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:07:52.839 INFO:teuthology.orchestra.run.smithi018.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:07:52.839 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:07:52.839 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-29T11:07:52.919 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:07:52.971 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:07:53.005 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:07:53.053 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:07:53.075 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:07:53.207 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:07:53.778 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:01:22 ago on Fri 29 Mar 2024 11:06:31 AM UTC. 2024-03-29T11:07:53.831 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:07:53.831 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:07:53.831 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:07:53.831 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:07:53.831 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:07:53.831 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:07:53.831 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:07:53.831 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:07:53.831 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:07:53.831 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:07:53.831 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:07:53.832 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:07:53.832 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:07:53.937 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-29T11:07:53.938 INFO:teuthology.orchestra.run.smithi018.stdout:======================================================================================== 2024-03-29T11:07:53.938 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repository Size 2024-03-29T11:07:53.939 INFO:teuthology.orchestra.run.smithi018.stdout:======================================================================================== 2024-03-29T11:07:53.939 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-29T11:07:53.939 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:07:53.939 INFO:teuthology.orchestra.run.smithi018.stdout:Installing dependencies: 2024-03-29T11:07:53.939 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:07:53.939 INFO:teuthology.orchestra.run.smithi018.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:07:53.939 INFO:teuthology.orchestra.run.smithi018.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:07:53.939 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:07:53.939 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:07:53.939 INFO:teuthology.orchestra.run.smithi018.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:07:53.939 INFO:teuthology.orchestra.run.smithi018.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:07:53.939 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:07:53.939 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:07:53.939 INFO:teuthology.orchestra.run.smithi018.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:07:53.940 INFO:teuthology.orchestra.run.smithi018.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:07:53.940 INFO:teuthology.orchestra.run.smithi018.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:07:53.940 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:07:53.940 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-29T11:07:53.940 INFO:teuthology.orchestra.run.smithi018.stdout:======================================================================================== 2024-03-29T11:07:53.940 INFO:teuthology.orchestra.run.smithi018.stdout:Install 13 Packages 2024-03-29T11:07:53.940 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:07:53.940 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 2.0 M 2024-03-29T11:07:53.941 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 26 M 2024-03-29T11:07:53.941 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-29T11:07:54.194 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:07:54.194 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:07:54.194 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-03-29T11:07:54.194 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:07:54.194 INFO:teuthology.orchestra.run.smithi070.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:07:54.195 INFO:teuthology.orchestra.run.smithi070.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:07:54.195 INFO:teuthology.orchestra.run.smithi070.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:07:54.195 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:07:54.195 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:07:54.195 INFO:teuthology.orchestra.run.smithi070.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:07:54.195 INFO:teuthology.orchestra.run.smithi070.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:07:54.195 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:07:54.195 INFO:teuthology.orchestra.run.smithi070.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:07:54.195 INFO:teuthology.orchestra.run.smithi070.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:07:54.195 INFO:teuthology.orchestra.run.smithi070.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:07:54.195 INFO:teuthology.orchestra.run.smithi070.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:07:54.195 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:07:54.196 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-03-29T11:07:54.267 INFO:teuthology.orchestra.run.smithi018.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 136 kB/s | 18 kB 00:00 2024-03-29T11:07:54.291 INFO:teuthology.orchestra.run.smithi018.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 155 kB/s | 24 kB 00:00 2024-03-29T11:07:54.350 INFO:teuthology.orchestra.run.smithi018.stdout:(3/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 241 kB/s | 52 kB 00:00 2024-03-29T11:07:54.375 INFO:teuthology.orchestra.run.smithi018.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.6 MB/s | 174 kB 00:00 2024-03-29T11:07:54.401 INFO:teuthology.orchestra.run.smithi018.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 132 kB 00:00 2024-03-29T11:07:54.426 INFO:teuthology.orchestra.run.smithi018.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 211 kB 00:00 2024-03-29T11:07:54.451 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:07:54.451 INFO:teuthology.orchestra.run.smithi018.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 569 kB/s | 43 kB 00:00 2024-03-29T11:07:54.568 INFO:teuthology.orchestra.run.smithi018.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 129 kB/s | 15 kB 00:00 2024-03-29T11:07:54.593 INFO:teuthology.orchestra.run.smithi018.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 167 kB/s | 32 kB 00:00 2024-03-29T11:07:54.702 INFO:teuthology.orchestra.run.smithi018.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 810 kB/s | 203 kB 00:00 2024-03-29T11:07:54.727 INFO:teuthology.orchestra.run.smithi018.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 443 kB/s | 59 kB 00:00 2024-03-29T11:07:54.794 INFO:teuthology.orchestra.run.smithi018.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 979 kB/s | 90 kB 00:00 2024-03-29T11:07:54.861 INFO:teuthology.orchestra.run.smithi018.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.5 MB/s | 1.0 MB 00:00 2024-03-29T11:07:54.863 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:54.863 INFO:teuthology.orchestra.run.smithi018.stdout:Total 2.2 MB/s | 2.0 MB 00:00 2024-03-29T11:07:54.863 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-29T11:07:54.878 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-29T11:07:54.878 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-29T11:07:54.977 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-29T11:07:54.978 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-29T11:07:55.019 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:22 ago on Fri 29 Mar 2024 11:06:33 AM UTC. 2024-03-29T11:07:55.172 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-03-29T11:07:55.172 INFO:teuthology.orchestra.run.smithi070.stdout:======================================================================================== 2024-03-29T11:07:55.172 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-03-29T11:07:55.172 INFO:teuthology.orchestra.run.smithi070.stdout:======================================================================================== 2024-03-29T11:07:55.173 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-03-29T11:07:55.173 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:07:55.173 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-03-29T11:07:55.173 INFO:teuthology.orchestra.run.smithi070.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:07:55.173 INFO:teuthology.orchestra.run.smithi070.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:07:55.173 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:07:55.173 INFO:teuthology.orchestra.run.smithi070.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:07:55.173 INFO:teuthology.orchestra.run.smithi070.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:07:55.173 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:07:55.173 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-03-29T11:07:55.173 INFO:teuthology.orchestra.run.smithi070.stdout:======================================================================================== 2024-03-29T11:07:55.173 INFO:teuthology.orchestra.run.smithi070.stdout:Install 6 Packages 2024-03-29T11:07:55.173 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:07:55.174 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 6.8 M 2024-03-29T11:07:55.174 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 33 M 2024-03-29T11:07:55.174 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-03-29T11:07:55.517 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-29T11:07:55.691 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:07:55.730 INFO:teuthology.orchestra.run.smithi070.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-03-29T11:07:55.747 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:07:55.756 INFO:teuthology.orchestra.run.smithi070.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 425 kB/s | 138 kB 00:00 2024-03-29T11:07:55.803 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:07:55.839 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:07:55.856 INFO:teuthology.orchestra.run.smithi070.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 584 kB/s | 248 kB 00:00 2024-03-29T11:07:55.873 INFO:teuthology.orchestra.run.smithi070.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 494 kB/s | 58 kB 00:00 2024-03-29T11:07:55.880 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:07:55.973 INFO:teuthology.orchestra.run.smithi070.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.2 MB/s | 548 kB 00:00 2024-03-29T11:07:55.978 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:07:55.990 INFO:teuthology.orchestra.run.smithi070.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 381 kB/s | 51 kB 00:00 2024-03-29T11:07:55.991 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:55.991 INFO:teuthology.orchestra.run.smithi070.stdout:Total 8.3 MB/s | 6.8 MB 00:00 2024-03-29T11:07:56.005 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-03-29T11:07:56.015 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-03-29T11:07:56.015 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-03-29T11:07:56.089 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:07:56.104 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-03-29T11:07:56.104 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-03-29T11:07:56.141 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:07:56.404 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-03-29T11:07:56.564 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:07:56.617 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:07:56.899 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:07:56.953 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:07:56.987 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:07:57.036 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:07:57.065 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:07:57.140 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:07:57.219 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:07:57.280 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:07:57.304 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:07:57.889 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:07:57.889 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:07:57.889 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:07:57.889 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:07:57.889 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:07:57.890 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:07:57.890 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:07:57.890 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:07:57.890 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:07:57.890 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:07:57.890 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:07:57.890 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:07:57.890 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:07:58.071 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:07:58.071 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:07:58.071 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:07:58.072 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:07:58.072 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:07:58.072 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:07:58.252 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:07:58.252 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:07:58.252 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-29T11:07:58.252 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:07:58.252 INFO:teuthology.orchestra.run.smithi018.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:07:58.252 INFO:teuthology.orchestra.run.smithi018.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:07:58.252 INFO:teuthology.orchestra.run.smithi018.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:07:58.252 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:07:58.252 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:07:58.253 INFO:teuthology.orchestra.run.smithi018.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:07:58.253 INFO:teuthology.orchestra.run.smithi018.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:07:58.253 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:07:58.253 INFO:teuthology.orchestra.run.smithi018.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:07:58.253 INFO:teuthology.orchestra.run.smithi018.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:07:58.253 INFO:teuthology.orchestra.run.smithi018.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:07:58.253 INFO:teuthology.orchestra.run.smithi018.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:07:58.253 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:07:58.253 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-29T11:07:58.376 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:07:58.376 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:07:58.376 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-03-29T11:07:58.376 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:07:58.376 INFO:teuthology.orchestra.run.smithi070.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:07:58.376 INFO:teuthology.orchestra.run.smithi070.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:07:58.376 INFO:teuthology.orchestra.run.smithi070.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:07:58.376 INFO:teuthology.orchestra.run.smithi070.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:07:58.376 INFO:teuthology.orchestra.run.smithi070.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:07:58.376 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:07:58.376 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-03-29T11:07:58.491 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:07:58.611 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-fuse 2024-03-29T11:07:59.069 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:01:28 ago on Fri 29 Mar 2024 11:06:31 AM UTC. 2024-03-29T11:07:59.174 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:26 ago on Fri 29 Mar 2024 11:06:33 AM UTC. 2024-03-29T11:07:59.227 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-29T11:07:59.228 INFO:teuthology.orchestra.run.smithi018.stdout:======================================================================================== 2024-03-29T11:07:59.228 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repository Size 2024-03-29T11:07:59.228 INFO:teuthology.orchestra.run.smithi018.stdout:======================================================================================== 2024-03-29T11:07:59.228 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-29T11:07:59.228 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:07:59.228 INFO:teuthology.orchestra.run.smithi018.stdout:Installing dependencies: 2024-03-29T11:07:59.228 INFO:teuthology.orchestra.run.smithi018.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:07:59.228 INFO:teuthology.orchestra.run.smithi018.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:07:59.228 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:07:59.229 INFO:teuthology.orchestra.run.smithi018.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:07:59.229 INFO:teuthology.orchestra.run.smithi018.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:07:59.229 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:07:59.229 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-29T11:07:59.229 INFO:teuthology.orchestra.run.smithi018.stdout:======================================================================================== 2024-03-29T11:07:59.229 INFO:teuthology.orchestra.run.smithi018.stdout:Install 6 Packages 2024-03-29T11:07:59.229 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:07:59.229 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 6.8 M 2024-03-29T11:07:59.230 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 33 M 2024-03-29T11:07:59.230 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-29T11:07:59.324 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-03-29T11:07:59.324 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-03-29T11:07:59.324 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-03-29T11:07:59.324 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-03-29T11:07:59.324 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-03-29T11:07:59.325 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:07:59.325 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-03-29T11:07:59.325 INFO:teuthology.orchestra.run.smithi070.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:07:59.325 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:07:59.325 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-03-29T11:07:59.325 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-03-29T11:07:59.325 INFO:teuthology.orchestra.run.smithi070.stdout:Install 2 Packages 2024-03-29T11:07:59.325 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:07:59.325 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 964 k 2024-03-29T11:07:59.326 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 2.8 M 2024-03-29T11:07:59.326 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-03-29T11:07:59.450 INFO:teuthology.orchestra.run.smithi070.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 636 kB/s | 79 kB 00:00 2024-03-29T11:07:59.641 INFO:teuthology.orchestra.run.smithi070.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 2.7 MB/s | 885 kB 00:00 2024-03-29T11:07:59.643 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:07:59.643 INFO:teuthology.orchestra.run.smithi070.stdout:Total 3.0 MB/s | 964 kB 00:00 2024-03-29T11:07:59.643 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-03-29T11:07:59.663 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-03-29T11:07:59.663 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-03-29T11:07:59.736 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-03-29T11:07:59.737 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-03-29T11:07:59.837 INFO:teuthology.orchestra.run.smithi018.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 5.8 MB 00:00 2024-03-29T11:07:59.896 INFO:teuthology.orchestra.run.smithi018.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 574 kB/s | 248 kB 00:00 2024-03-29T11:07:59.919 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-03-29T11:07:59.921 INFO:teuthology.orchestra.run.smithi018.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 302 kB/s | 138 kB 00:00 2024-03-29T11:08:00.047 INFO:teuthology.orchestra.run.smithi018.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 384 kB/s | 58 kB 00:00 2024-03-29T11:08:00.068 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:08:00.072 INFO:teuthology.orchestra.run.smithi018.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 339 kB/s | 51 kB 00:00 2024-03-29T11:08:00.105 INFO:teuthology.orchestra.run.smithi018.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.0 MB/s | 548 kB 00:00 2024-03-29T11:08:00.107 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:08:00.108 INFO:teuthology.orchestra.run.smithi018.stdout:Total 7.7 MB/s | 6.8 MB 00:00 2024-03-29T11:08:00.121 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-29T11:08:00.132 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-29T11:08:00.132 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-29T11:08:00.141 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:08:00.223 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-29T11:08:00.223 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-29T11:08:00.522 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-29T11:08:00.705 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:08:00.758 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:08:00.801 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:08:00.801 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:08:01.093 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:08:01.093 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:01.093 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-03-29T11:08:01.093 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:08:01.093 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:01.093 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-03-29T11:08:01.315 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:08:01.362 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install ceph-volume 2024-03-29T11:08:01.413 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:08:01.482 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:08:01.511 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:08:01.941 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:28 ago on Fri 29 Mar 2024 11:06:33 AM UTC. 2024-03-29T11:08:02.097 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-03-29T11:08:02.097 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-03-29T11:08:02.097 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repository Size 2024-03-29T11:08:02.098 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-03-29T11:08:02.098 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-03-29T11:08:02.098 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:08:02.098 INFO:teuthology.orchestra.run.smithi070.stdout:Installing dependencies: 2024-03-29T11:08:02.098 INFO:teuthology.orchestra.run.smithi070.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:08:02.098 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:02.098 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-03-29T11:08:02.098 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-03-29T11:08:02.098 INFO:teuthology.orchestra.run.smithi070.stdout:Install 2 Packages 2024-03-29T11:08:02.098 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:02.099 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 332 k 2024-03-29T11:08:02.099 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 1.5 M 2024-03-29T11:08:02.099 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-03-29T11:08:02.238 INFO:teuthology.orchestra.run.smithi070.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 493 kB/s | 69 kB 00:00 2024-03-29T11:08:02.456 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:08:02.456 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:08:02.456 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:08:02.456 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:08:02.457 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:08:02.457 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:08:02.596 INFO:teuthology.orchestra.run.smithi070.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 530 kB/s | 264 kB 00:00 2024-03-29T11:08:02.597 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:08:02.598 INFO:teuthology.orchestra.run.smithi070.stdout:Total 665 kB/s | 332 kB 00:00 2024-03-29T11:08:02.598 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-03-29T11:08:02.607 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-03-29T11:08:02.607 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-03-29T11:08:02.647 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-03-29T11:08:02.647 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-03-29T11:08:02.758 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:08:02.758 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:02.758 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-29T11:08:02.758 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:08:02.758 INFO:teuthology.orchestra.run.smithi018.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:08:02.758 INFO:teuthology.orchestra.run.smithi018.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:08:02.758 INFO:teuthology.orchestra.run.smithi018.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:08:02.759 INFO:teuthology.orchestra.run.smithi018.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:08:02.759 INFO:teuthology.orchestra.run.smithi018.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:08:02.759 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:02.759 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-29T11:08:02.802 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-03-29T11:08:02.978 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:08:02.990 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-fuse 2024-03-29T11:08:03.024 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:08:03.040 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:08:03.041 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:08:03.041 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:08:03.041 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:03.556 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:01:32 ago on Fri 29 Mar 2024 11:06:31 AM UTC. 2024-03-29T11:08:03.713 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-29T11:08:03.713 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-29T11:08:03.713 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repository Size 2024-03-29T11:08:03.713 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-29T11:08:03.713 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-29T11:08:03.713 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:08:03.714 INFO:teuthology.orchestra.run.smithi018.stdout:Installing dependencies: 2024-03-29T11:08:03.714 INFO:teuthology.orchestra.run.smithi018.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:08:03.714 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:03.714 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-29T11:08:03.714 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-29T11:08:03.714 INFO:teuthology.orchestra.run.smithi018.stdout:Install 2 Packages 2024-03-29T11:08:03.714 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:03.715 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 964 k 2024-03-29T11:08:03.715 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 2.8 M 2024-03-29T11:08:03.715 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-29T11:08:03.830 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:08:03.840 INFO:teuthology.orchestra.run.smithi018.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 627 kB/s | 79 kB 00:00 2024-03-29T11:08:04.169 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:08:04.169 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:04.169 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-03-29T11:08:04.169 INFO:teuthology.orchestra.run.smithi070.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:08:04.169 INFO:teuthology.orchestra.run.smithi070.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:08:04.170 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:04.170 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-03-29T11:08:04.398 INFO:teuthology.orchestra.run.smithi018.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.3 MB/s | 885 kB 00:00 2024-03-29T11:08:04.399 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:08:04.400 INFO:teuthology.orchestra.run.smithi018.stdout:Total 1.4 MB/s | 964 kB 00:00 2024-03-29T11:08:04.400 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-29T11:08:04.423 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-29T11:08:04.423 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-29T11:08:04.493 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install librados-devel 2024-03-29T11:08:04.500 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-29T11:08:04.506 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-29T11:08:04.687 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-29T11:08:04.829 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:08:04.910 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:08:05.068 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:32 ago on Fri 29 Mar 2024 11:06:33 AM UTC. 2024-03-29T11:08:05.220 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-03-29T11:08:05.221 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-03-29T11:08:05.221 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repo Size 2024-03-29T11:08:05.221 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-03-29T11:08:05.221 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-03-29T11:08:05.221 INFO:teuthology.orchestra.run.smithi070.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:08:05.221 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:05.221 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-03-29T11:08:05.221 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-03-29T11:08:05.221 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2024-03-29T11:08:05.221 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:05.222 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 127 k 2024-03-29T11:08:05.222 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 456 k 2024-03-29T11:08:05.222 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-03-29T11:08:05.538 INFO:teuthology.orchestra.run.smithi070.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 401 kB/s | 127 kB 00:00 2024-03-29T11:08:05.539 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:08:05.539 INFO:teuthology.orchestra.run.smithi070.stdout:Total 399 kB/s | 127 kB 00:00 2024-03-29T11:08:05.539 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-03-29T11:08:05.545 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-03-29T11:08:05.545 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-03-29T11:08:05.588 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:08:05.588 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:08:05.602 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-03-29T11:08:05.602 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-03-29T11:08:05.748 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-03-29T11:08:05.846 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:08:05.884 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:08:05.884 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:05.885 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-29T11:08:05.885 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:08:05.885 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:05.885 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-29T11:08:06.146 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install ceph-volume 2024-03-29T11:08:06.297 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:08:06.600 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:08:06.600 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:06.600 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-03-29T11:08:06.600 INFO:teuthology.orchestra.run.smithi070.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:08:06.600 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:06.600 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-03-29T11:08:06.728 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:01:35 ago on Fri 29 Mar 2024 11:06:31 AM UTC. 2024-03-29T11:08:06.843 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install libcephfs2 2024-03-29T11:08:06.886 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-29T11:08:06.887 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-29T11:08:06.887 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repository Size 2024-03-29T11:08:06.887 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-29T11:08:06.887 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-29T11:08:06.887 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:08:06.887 INFO:teuthology.orchestra.run.smithi018.stdout:Installing dependencies: 2024-03-29T11:08:06.888 INFO:teuthology.orchestra.run.smithi018.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:08:06.888 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:06.888 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-29T11:08:06.888 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-29T11:08:06.888 INFO:teuthology.orchestra.run.smithi018.stdout:Install 2 Packages 2024-03-29T11:08:06.888 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:06.889 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 332 k 2024-03-29T11:08:06.889 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 1.5 M 2024-03-29T11:08:06.889 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-29T11:08:07.014 INFO:teuthology.orchestra.run.smithi018.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 552 kB/s | 69 kB 00:00 2024-03-29T11:08:07.189 INFO:teuthology.orchestra.run.smithi018.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 880 kB/s | 264 kB 00:00 2024-03-29T11:08:07.190 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:08:07.190 INFO:teuthology.orchestra.run.smithi018.stdout:Total 1.1 MB/s | 332 kB 00:00 2024-03-29T11:08:07.190 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-29T11:08:07.199 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-29T11:08:07.200 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-29T11:08:07.240 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-29T11:08:07.240 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-29T11:08:07.397 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-29T11:08:07.423 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:34 ago on Fri 29 Mar 2024 11:06:33 AM UTC. 2024-03-29T11:08:07.521 INFO:teuthology.orchestra.run.smithi070.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:08:07.569 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:08:07.574 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-03-29T11:08:07.575 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-03-29T11:08:07.575 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-03-29T11:08:07.600 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:08:07.616 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:08:07.616 INFO:teuthology.orchestra.run.smithi018.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:08:07.616 INFO:teuthology.orchestra.run.smithi018.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:08:07.616 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:07.641 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install libcephfs-devel 2024-03-29T11:08:08.218 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:35 ago on Fri 29 Mar 2024 11:06:33 AM UTC. 2024-03-29T11:08:08.309 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:08:08.368 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-03-29T11:08:08.369 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-03-29T11:08:08.369 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repo Size 2024-03-29T11:08:08.369 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-03-29T11:08:08.369 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-03-29T11:08:08.369 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:08:08.369 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:08.369 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-03-29T11:08:08.369 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-03-29T11:08:08.369 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2024-03-29T11:08:08.369 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:08.370 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 31 k 2024-03-29T11:08:08.370 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 140 k 2024-03-29T11:08:08.370 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-03-29T11:08:08.625 INFO:teuthology.orchestra.run.smithi070.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 122 kB/s | 31 kB 00:00 2024-03-29T11:08:08.625 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:08:08.625 INFO:teuthology.orchestra.run.smithi070.stdout:Total 122 kB/s | 31 kB 00:00 2024-03-29T11:08:08.625 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-03-29T11:08:08.629 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-03-29T11:08:08.629 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-03-29T11:08:08.636 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:08:08.636 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:08.636 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-29T11:08:08.636 INFO:teuthology.orchestra.run.smithi018.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:08:08.636 INFO:teuthology.orchestra.run.smithi018.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:08:08.636 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:08.636 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-29T11:08:08.654 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-03-29T11:08:08.655 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-03-29T11:08:08.785 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-03-29T11:08:08.898 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:08:08.956 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install librados-devel 2024-03-29T11:08:09.305 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:08:09.535 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Mar 2024 11:06:31 AM UTC. 2024-03-29T11:08:09.566 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:08:09.566 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:09.566 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-03-29T11:08:09.566 INFO:teuthology.orchestra.run.smithi070.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:08:09.566 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:09.566 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-03-29T11:08:09.692 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-29T11:08:09.692 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-29T11:08:09.693 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-29T11:08:09.693 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-29T11:08:09.693 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-29T11:08:09.693 INFO:teuthology.orchestra.run.smithi018.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:08:09.693 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:09.693 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-29T11:08:09.693 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-29T11:08:09.693 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-29T11:08:09.693 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:09.693 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 127 k 2024-03-29T11:08:09.693 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 456 k 2024-03-29T11:08:09.694 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-29T11:08:09.743 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install librados2 2024-03-29T11:08:09.886 INFO:teuthology.orchestra.run.smithi018.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 656 kB/s | 127 kB 00:00 2024-03-29T11:08:09.887 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:08:09.887 INFO:teuthology.orchestra.run.smithi018.stdout:Total 651 kB/s | 127 kB 00:00 2024-03-29T11:08:09.887 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-29T11:08:09.893 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-29T11:08:09.893 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-29T11:08:09.951 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-29T11:08:09.952 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-29T11:08:10.101 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-29T11:08:10.205 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:08:10.305 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:37 ago on Fri 29 Mar 2024 11:06:33 AM UTC. 2024-03-29T11:08:10.402 INFO:teuthology.orchestra.run.smithi070.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:08:10.454 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-03-29T11:08:10.455 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-03-29T11:08:10.455 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-03-29T11:08:10.496 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install librbd1 2024-03-29T11:08:10.637 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:08:10.930 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:08:10.930 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:10.930 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-29T11:08:10.930 INFO:teuthology.orchestra.run.smithi018.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:08:10.930 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:10.930 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-29T11:08:11.057 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Mar 2024 11:06:33 AM UTC. 2024-03-29T11:08:11.153 INFO:teuthology.orchestra.run.smithi070.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:08:11.171 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install libcephfs2 2024-03-29T11:08:11.206 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-03-29T11:08:11.207 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-03-29T11:08:11.207 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-03-29T11:08:11.253 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install python3-rados 2024-03-29T11:08:11.808 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 11:06:31 AM UTC. 2024-03-29T11:08:11.821 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:38 ago on Fri 29 Mar 2024 11:06:33 AM UTC. 2024-03-29T11:08:11.910 INFO:teuthology.orchestra.run.smithi018.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:08:11.920 INFO:teuthology.orchestra.run.smithi070.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:08:11.967 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-29T11:08:11.968 INFO:teuthology.orchestra.run.smithi018.stdout:Nothing to do. 2024-03-29T11:08:11.968 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-29T11:08:11.972 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-03-29T11:08:11.973 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-03-29T11:08:11.973 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-03-29T11:08:12.016 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install python3-rgw 2024-03-29T11:08:12.044 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install libcephfs-devel 2024-03-29T11:08:12.574 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 11:06:33 AM UTC. 2024-03-29T11:08:12.630 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 11:06:31 AM UTC. 2024-03-29T11:08:12.670 INFO:teuthology.orchestra.run.smithi070.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:08:12.721 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-03-29T11:08:12.722 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-03-29T11:08:12.723 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-03-29T11:08:12.774 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install python3-cephfs 2024-03-29T11:08:12.789 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-29T11:08:12.789 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-29T11:08:12.789 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-29T11:08:12.789 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-29T11:08:12.790 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-29T11:08:12.790 INFO:teuthology.orchestra.run.smithi018.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:08:12.790 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:12.790 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-29T11:08:12.790 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-29T11:08:12.790 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-29T11:08:12.790 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:12.790 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 31 k 2024-03-29T11:08:12.791 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 140 k 2024-03-29T11:08:12.791 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-29T11:08:13.057 INFO:teuthology.orchestra.run.smithi018.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 117 kB/s | 31 kB 00:00 2024-03-29T11:08:13.058 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:08:13.058 INFO:teuthology.orchestra.run.smithi018.stdout:Total 116 kB/s | 31 kB 00:00 2024-03-29T11:08:13.058 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-29T11:08:13.062 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-29T11:08:13.062 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-29T11:08:13.092 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-29T11:08:13.092 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-29T11:08:13.217 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-29T11:08:13.333 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:08:13.342 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 11:06:33 AM UTC. 2024-03-29T11:08:13.441 INFO:teuthology.orchestra.run.smithi070.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:08:13.494 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-03-29T11:08:13.495 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-03-29T11:08:13.495 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-03-29T11:08:13.537 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install python3-rbd 2024-03-29T11:08:13.689 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:08:13.946 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:08:13.946 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:13.947 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-29T11:08:13.947 INFO:teuthology.orchestra.run.smithi018.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:08:13.947 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:13.947 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-29T11:08:14.100 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install librados2 2024-03-29T11:08:14.101 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 11:06:33 AM UTC. 2024-03-29T11:08:14.198 INFO:teuthology.orchestra.run.smithi070.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:08:14.250 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-03-29T11:08:14.277 INFO:teuthology.orchestra.run.smithi070.stdout:Nothing to do. 2024-03-29T11:08:14.277 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-03-29T11:08:14.295 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install rbd-fuse 2024-03-29T11:08:14.670 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 11:06:31 AM UTC. 2024-03-29T11:08:14.770 INFO:teuthology.orchestra.run.smithi018.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:08:14.825 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-29T11:08:14.826 INFO:teuthology.orchestra.run.smithi018.stdout:Nothing to do. 2024-03-29T11:08:14.827 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-29T11:08:14.856 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 11:06:33 AM UTC. 2024-03-29T11:08:14.871 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install librbd1 2024-03-29T11:08:15.003 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-03-29T11:08:15.004 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-03-29T11:08:15.004 INFO:teuthology.orchestra.run.smithi070.stdout: Package Architecture Version Repository Size 2024-03-29T11:08:15.004 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-03-29T11:08:15.004 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-03-29T11:08:15.004 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:08:15.004 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:15.004 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-03-29T11:08:15.005 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-03-29T11:08:15.005 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2024-03-29T11:08:15.005 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:15.005 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 85 k 2024-03-29T11:08:15.005 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 231 k 2024-03-29T11:08:15.005 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-03-29T11:08:15.294 INFO:teuthology.orchestra.run.smithi070.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 295 kB/s | 85 kB 00:00 2024-03-29T11:08:15.295 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:08:15.295 INFO:teuthology.orchestra.run.smithi070.stdout:Total 294 kB/s | 85 kB 00:00 2024-03-29T11:08:15.295 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-03-29T11:08:15.302 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-03-29T11:08:15.302 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-03-29T11:08:15.358 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-03-29T11:08:15.358 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-03-29T11:08:15.444 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 11:06:31 AM UTC. 2024-03-29T11:08:15.498 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-03-29T11:08:15.545 INFO:teuthology.orchestra.run.smithi018.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:08:15.601 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-29T11:08:15.601 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:08:15.602 INFO:teuthology.orchestra.run.smithi018.stdout:Nothing to do. 2024-03-29T11:08:15.602 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-29T11:08:15.652 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install python3-rados 2024-03-29T11:08:15.994 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:08:16.224 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 11:06:31 AM UTC. 2024-03-29T11:08:16.302 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:08:16.302 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:16.302 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-03-29T11:08:16.302 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:08:16.302 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:16.302 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-03-29T11:08:16.325 INFO:teuthology.orchestra.run.smithi018.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:08:16.380 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-29T11:08:16.381 INFO:teuthology.orchestra.run.smithi018.stdout:Nothing to do. 2024-03-29T11:08:16.381 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-29T11:08:16.426 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install python3-rgw 2024-03-29T11:08:16.539 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install rbd-mirror 2024-03-29T11:08:17.007 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 11:06:31 AM UTC. 2024-03-29T11:08:17.107 INFO:teuthology.orchestra.run.smithi018.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:08:17.118 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 11:06:33 AM UTC. 2024-03-29T11:08:17.162 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-29T11:08:17.163 INFO:teuthology.orchestra.run.smithi018.stdout:Nothing to do. 2024-03-29T11:08:17.164 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-29T11:08:17.210 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install python3-cephfs 2024-03-29T11:08:17.273 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-03-29T11:08:17.274 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-03-29T11:08:17.274 INFO:teuthology.orchestra.run.smithi070.stdout: Package Arch Version Repo Size 2024-03-29T11:08:17.274 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-03-29T11:08:17.274 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-03-29T11:08:17.274 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:08:17.274 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:17.274 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-03-29T11:08:17.274 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-03-29T11:08:17.274 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2024-03-29T11:08:17.274 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:17.274 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 3.1 M 2024-03-29T11:08:17.275 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 13 M 2024-03-29T11:08:17.275 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-03-29T11:08:17.789 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 11:06:31 AM UTC. 2024-03-29T11:08:17.891 INFO:teuthology.orchestra.run.smithi018.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:08:17.945 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-29T11:08:17.946 INFO:teuthology.orchestra.run.smithi018.stdout:Nothing to do. 2024-03-29T11:08:17.946 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-29T11:08:17.988 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install python3-rbd 2024-03-29T11:08:17.990 INFO:teuthology.orchestra.run.smithi070.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 4.3 MB/s | 3.1 MB 00:00 2024-03-29T11:08:17.991 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:08:17.991 INFO:teuthology.orchestra.run.smithi070.stdout:Total 4.3 MB/s | 3.1 MB 00:00 2024-03-29T11:08:17.991 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-03-29T11:08:18.002 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-03-29T11:08:18.002 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-03-29T11:08:18.078 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-03-29T11:08:18.078 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-03-29T11:08:18.513 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-03-29T11:08:18.564 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 11:06:31 AM UTC. 2024-03-29T11:08:18.575 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:08:18.599 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:08:18.599 INFO:teuthology.orchestra.run.smithi070.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:08:18.599 INFO:teuthology.orchestra.run.smithi070.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:08:18.599 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:08:18.600 INFO:teuthology.orchestra.run.smithi070.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:08:18.600 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:18.665 INFO:teuthology.orchestra.run.smithi018.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:08:18.720 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-29T11:08:18.721 INFO:teuthology.orchestra.run.smithi018.stdout:Nothing to do. 2024-03-29T11:08:18.721 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-29T11:08:18.766 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install rbd-fuse 2024-03-29T11:08:19.337 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:01:48 ago on Fri 29 Mar 2024 11:06:31 AM UTC. 2024-03-29T11:08:19.498 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-29T11:08:19.498 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-29T11:08:19.498 INFO:teuthology.orchestra.run.smithi018.stdout: Package Architecture Version Repository Size 2024-03-29T11:08:19.498 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-29T11:08:19.498 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-29T11:08:19.498 INFO:teuthology.orchestra.run.smithi018.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:08:19.498 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:19.498 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-29T11:08:19.498 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-29T11:08:19.499 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-29T11:08:19.499 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:19.499 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 85 k 2024-03-29T11:08:19.499 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 231 k 2024-03-29T11:08:19.499 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-29T11:08:19.702 INFO:teuthology.orchestra.run.smithi018.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 407 kB/s | 85 kB 00:00 2024-03-29T11:08:19.708 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:08:19.708 INFO:teuthology.orchestra.run.smithi018.stdout:Total 405 kB/s | 85 kB 00:00 2024-03-29T11:08:19.708 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-29T11:08:19.711 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-29T11:08:19.711 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-29T11:08:19.770 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-29T11:08:19.770 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-29T11:08:19.806 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:08:19.807 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:19.808 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-03-29T11:08:19.808 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:08:19.809 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:19.809 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-03-29T11:08:19.914 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-29T11:08:20.028 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:08:20.052 DEBUG:teuthology.orchestra.run.smithi070:> sudo yum -y install rbd-nbd 2024-03-29T11:08:20.407 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:08:20.627 INFO:teuthology.orchestra.run.smithi070.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 11:06:33 AM UTC. 2024-03-29T11:08:20.702 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:08:20.702 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:20.703 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-29T11:08:20.703 INFO:teuthology.orchestra.run.smithi018.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:08:20.703 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:20.703 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-29T11:08:20.776 INFO:teuthology.orchestra.run.smithi070.stdout:Dependencies resolved. 2024-03-29T11:08:20.777 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-03-29T11:08:20.777 INFO:teuthology.orchestra.run.smithi070.stdout: Package Architecture Version Repository Size 2024-03-29T11:08:20.777 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-03-29T11:08:20.777 INFO:teuthology.orchestra.run.smithi070.stdout:Installing: 2024-03-29T11:08:20.777 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:08:20.777 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:20.777 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction Summary 2024-03-29T11:08:20.777 INFO:teuthology.orchestra.run.smithi070.stdout:================================================================================ 2024-03-29T11:08:20.777 INFO:teuthology.orchestra.run.smithi070.stdout:Install 1 Package 2024-03-29T11:08:20.777 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:20.777 INFO:teuthology.orchestra.run.smithi070.stdout:Total download size: 171 k 2024-03-29T11:08:20.777 INFO:teuthology.orchestra.run.smithi070.stdout:Installed size: 494 k 2024-03-29T11:08:20.778 INFO:teuthology.orchestra.run.smithi070.stdout:Downloading Packages: 2024-03-29T11:08:20.972 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install rbd-mirror 2024-03-29T11:08:21.185 INFO:teuthology.orchestra.run.smithi070.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 421 kB/s | 171 kB 00:00 2024-03-29T11:08:21.186 INFO:teuthology.orchestra.run.smithi070.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:08:21.186 INFO:teuthology.orchestra.run.smithi070.stdout:Total 419 kB/s | 171 kB 00:00 2024-03-29T11:08:21.186 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction check 2024-03-29T11:08:21.195 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction check succeeded. 2024-03-29T11:08:21.195 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction test 2024-03-29T11:08:21.250 INFO:teuthology.orchestra.run.smithi070.stdout:Transaction test succeeded. 2024-03-29T11:08:21.251 INFO:teuthology.orchestra.run.smithi070.stdout:Running transaction 2024-03-29T11:08:21.401 INFO:teuthology.orchestra.run.smithi070.stdout: Preparing : 1/1 2024-03-29T11:08:21.517 INFO:teuthology.orchestra.run.smithi070.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:08:21.554 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:01:50 ago on Fri 29 Mar 2024 11:06:31 AM UTC. 2024-03-29T11:08:21.714 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-29T11:08:21.715 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-29T11:08:21.715 INFO:teuthology.orchestra.run.smithi018.stdout: Package Arch Version Repo Size 2024-03-29T11:08:21.715 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-29T11:08:21.715 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-29T11:08:21.715 INFO:teuthology.orchestra.run.smithi018.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:08:21.715 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:21.715 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-29T11:08:21.715 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-29T11:08:21.715 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-29T11:08:21.715 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:21.715 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 3.1 M 2024-03-29T11:08:21.715 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 13 M 2024-03-29T11:08:21.715 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-29T11:08:21.958 INFO:teuthology.orchestra.run.smithi070.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:08:22.247 INFO:teuthology.orchestra.run.smithi070.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:08:22.247 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:22.247 INFO:teuthology.orchestra.run.smithi070.stdout:Installed: 2024-03-29T11:08:22.247 INFO:teuthology.orchestra.run.smithi070.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:08:22.247 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:22.247 INFO:teuthology.orchestra.run.smithi070.stdout:Complete! 2024-03-29T11:08:22.330 INFO:teuthology.orchestra.run.smithi018.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 5.0 MB/s | 3.1 MB 00:00 2024-03-29T11:08:22.331 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:08:22.331 INFO:teuthology.orchestra.run.smithi018.stdout:Total 5.0 MB/s | 3.1 MB 00:00 2024-03-29T11:08:22.332 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-29T11:08:22.342 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-29T11:08:22.342 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-29T11:08:22.420 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-29T11:08:22.420 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-29T11:08:22.480 DEBUG:teuthology.parallel:result is None 2024-03-29T11:08:22.873 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-29T11:08:22.923 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:08:22.947 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:08:22.947 INFO:teuthology.orchestra.run.smithi018.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:08:22.947 INFO:teuthology.orchestra.run.smithi018.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:08:22.947 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:08:22.948 INFO:teuthology.orchestra.run.smithi018.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:08:22.948 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:24.069 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:08:24.069 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:24.069 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-29T11:08:24.069 INFO:teuthology.orchestra.run.smithi018.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:08:24.069 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:24.069 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-29T11:08:24.301 DEBUG:teuthology.orchestra.run.smithi018:> sudo yum -y install rbd-nbd 2024-03-29T11:08:24.876 INFO:teuthology.orchestra.run.smithi018.stdout:Last metadata expiration check: 0:01:53 ago on Fri 29 Mar 2024 11:06:31 AM UTC. 2024-03-29T11:08:25.033 INFO:teuthology.orchestra.run.smithi018.stdout:Dependencies resolved. 2024-03-29T11:08:25.034 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-29T11:08:25.034 INFO:teuthology.orchestra.run.smithi018.stdout: Package Architecture Version Repository Size 2024-03-29T11:08:25.034 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-29T11:08:25.034 INFO:teuthology.orchestra.run.smithi018.stdout:Installing: 2024-03-29T11:08:25.034 INFO:teuthology.orchestra.run.smithi018.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:08:25.034 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:25.035 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction Summary 2024-03-29T11:08:25.035 INFO:teuthology.orchestra.run.smithi018.stdout:================================================================================ 2024-03-29T11:08:25.035 INFO:teuthology.orchestra.run.smithi018.stdout:Install 1 Package 2024-03-29T11:08:25.035 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:25.035 INFO:teuthology.orchestra.run.smithi018.stdout:Total download size: 171 k 2024-03-29T11:08:25.036 INFO:teuthology.orchestra.run.smithi018.stdout:Installed size: 494 k 2024-03-29T11:08:25.036 INFO:teuthology.orchestra.run.smithi018.stdout:Downloading Packages: 2024-03-29T11:08:25.277 INFO:teuthology.orchestra.run.smithi018.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 710 kB/s | 171 kB 00:00 2024-03-29T11:08:25.277 INFO:teuthology.orchestra.run.smithi018.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:08:25.277 INFO:teuthology.orchestra.run.smithi018.stdout:Total 704 kB/s | 171 kB 00:00 2024-03-29T11:08:25.277 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction check 2024-03-29T11:08:25.285 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction check succeeded. 2024-03-29T11:08:25.285 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction test 2024-03-29T11:08:25.343 INFO:teuthology.orchestra.run.smithi018.stdout:Transaction test succeeded. 2024-03-29T11:08:25.343 INFO:teuthology.orchestra.run.smithi018.stdout:Running transaction 2024-03-29T11:08:25.499 INFO:teuthology.orchestra.run.smithi018.stdout: Preparing : 1/1 2024-03-29T11:08:25.611 INFO:teuthology.orchestra.run.smithi018.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:08:26.022 INFO:teuthology.orchestra.run.smithi018.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:08:26.330 INFO:teuthology.orchestra.run.smithi018.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:08:26.330 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:26.330 INFO:teuthology.orchestra.run.smithi018.stdout:Installed: 2024-03-29T11:08:26.330 INFO:teuthology.orchestra.run.smithi018.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:08:26.330 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:26.330 INFO:teuthology.orchestra.run.smithi018.stdout:Complete! 2024-03-29T11:08:26.561 DEBUG:teuthology.parallel:result is None 2024-03-29T11:08:26.561 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:08:26.709 DEBUG:teuthology.orchestra.run.smithi018:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:08:26.730 INFO:teuthology.orchestra.run.smithi018.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:08:26.730 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:08:26.730 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:08:26.732 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:08:26.978 DEBUG:teuthology.orchestra.run.smithi070:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:08:27.002 INFO:teuthology.orchestra.run.smithi070.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:08:27.002 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:08:27.003 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:08:27.005 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T11:08:27.005 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-29T11:08:27.005 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:08:27.034 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-03-29T11:08:27.034 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:08:27.073 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T11:08:27.073 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-29T11:08:27.073 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:08:27.101 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:08:27.168 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-03-29T11:08:27.169 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:08:27.199 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:08:27.269 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T11:08:27.269 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-29T11:08:27.269 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:08:27.296 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:08:27.362 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-03-29T11:08:27.363 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:08:27.390 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:08:27.457 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T11:08:27.458 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-29T11:08:27.458 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:08:27.485 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:08:27.552 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-03-29T11:08:27.552 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:08:27.579 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:08:27.646 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T11:08:27.726 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T11:08:27.727 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod 777 /var/log/ceph 2024-03-29T11:08:27.729 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 777 /var/log/ceph 2024-03-29T11:08:27.756 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T11:08:27.756 DEBUG:teuthology.orchestra.run.smithi018:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:08:27.797 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:08:27.824 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T11:08:27.824 DEBUG:teuthology.orchestra.run.smithi018:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:08:27.863 DEBUG:teuthology.orchestra.run.smithi070:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:08:27.899 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T11:08:27.900 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T11:08:27.900 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622677', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622677', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi018.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIahkFEMlmnbD1gAQSJHalvyf71pZc2mxw+81x+ayQCdypgeL+kns4j8XKqBYveluo9sNsk0UJX9n5RTUJZ0Mnk=', 'smithi070.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJrL8s29ynEsIThArj/rDyBt8CR2iFx2u+RvNeCjnlMFTVVtqNRzhhtNViVWYO5QIGCiXPV536ehJiwSbhIipA0='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T11:08:27.900 DEBUG:teuthology.orchestra.run.smithi018:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:08:27.931 DEBUG:teuthology.orchestra.run.smithi070:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:08:27.945 DEBUG:teuthology.orchestra.run.smithi018:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:08:27.986 DEBUG:teuthology.orchestra.run.smithi070:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:08:28.012 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-29T11:08:28.012 DEBUG:teuthology.orchestra.run.smithi018:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:08:28.065 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:08:28.065 DEBUG:teuthology.orchestra.run.smithi018:> stat /dev/vg_nvme/lv_1 2024-03-29T11:08:28.119 INFO:teuthology.orchestra.run.smithi018.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:08:28.120 INFO:teuthology.orchestra.run.smithi018.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:08:28.120 INFO:teuthology.orchestra.run.smithi018.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-29T11:08:28.120 INFO:teuthology.orchestra.run.smithi018.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:08:28.120 INFO:teuthology.orchestra.run.smithi018.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:08:28.120 INFO:teuthology.orchestra.run.smithi018.stdout:Access: 2024-03-29 11:08:23.216818364 +0000 2024-03-29T11:08:28.120 INFO:teuthology.orchestra.run.smithi018.stdout:Modify: 2024-03-29 11:05:48.435473323 +0000 2024-03-29T11:08:28.120 INFO:teuthology.orchestra.run.smithi018.stdout:Change: 2024-03-29 11:05:48.436473311 +0000 2024-03-29T11:08:28.120 INFO:teuthology.orchestra.run.smithi018.stdout: Birth: 2024-03-29 11:05:48.435473323 +0000 2024-03-29T11:08:28.120 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:08:28.185 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records in 2024-03-29T11:08:28.186 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records out 2024-03-29T11:08:28.186 INFO:teuthology.orchestra.run.smithi018.stderr:512 bytes copied, 9.2215e-05 s, 5.6 MB/s 2024-03-29T11:08:28.187 DEBUG:teuthology.orchestra.run.smithi018:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:08:28.243 DEBUG:teuthology.orchestra.run.smithi018:> stat /dev/vg_nvme/lv_2 2024-03-29T11:08:28.297 INFO:teuthology.orchestra.run.smithi018.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:08:28.298 INFO:teuthology.orchestra.run.smithi018.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:08:28.298 INFO:teuthology.orchestra.run.smithi018.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-29T11:08:28.298 INFO:teuthology.orchestra.run.smithi018.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:08:28.298 INFO:teuthology.orchestra.run.smithi018.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:08:28.298 INFO:teuthology.orchestra.run.smithi018.stdout:Access: 2024-03-29 11:08:23.216818364 +0000 2024-03-29T11:08:28.298 INFO:teuthology.orchestra.run.smithi018.stdout:Modify: 2024-03-29 11:05:48.785469154 +0000 2024-03-29T11:08:28.298 INFO:teuthology.orchestra.run.smithi018.stdout:Change: 2024-03-29 11:05:48.785469154 +0000 2024-03-29T11:08:28.298 INFO:teuthology.orchestra.run.smithi018.stdout: Birth: 2024-03-29 11:05:48.785469154 +0000 2024-03-29T11:08:28.298 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:08:28.363 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records in 2024-03-29T11:08:28.363 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records out 2024-03-29T11:08:28.363 INFO:teuthology.orchestra.run.smithi018.stderr:512 bytes copied, 9.2858e-05 s, 5.5 MB/s 2024-03-29T11:08:28.364 DEBUG:teuthology.orchestra.run.smithi018:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:08:28.420 DEBUG:teuthology.orchestra.run.smithi018:> stat /dev/vg_nvme/lv_3 2024-03-29T11:08:28.473 INFO:teuthology.orchestra.run.smithi018.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:08:28.473 INFO:teuthology.orchestra.run.smithi018.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:08:28.474 INFO:teuthology.orchestra.run.smithi018.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-29T11:08:28.474 INFO:teuthology.orchestra.run.smithi018.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:08:28.474 INFO:teuthology.orchestra.run.smithi018.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:08:28.474 INFO:teuthology.orchestra.run.smithi018.stdout:Access: 2024-03-29 11:08:23.217818352 +0000 2024-03-29T11:08:28.474 INFO:teuthology.orchestra.run.smithi018.stdout:Modify: 2024-03-29 11:05:49.133465008 +0000 2024-03-29T11:08:28.474 INFO:teuthology.orchestra.run.smithi018.stdout:Change: 2024-03-29 11:05:49.133465008 +0000 2024-03-29T11:08:28.474 INFO:teuthology.orchestra.run.smithi018.stdout: Birth: 2024-03-29 11:05:49.133465008 +0000 2024-03-29T11:08:28.474 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:08:28.538 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records in 2024-03-29T11:08:28.538 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records out 2024-03-29T11:08:28.538 INFO:teuthology.orchestra.run.smithi018.stderr:512 bytes copied, 9.7565e-05 s, 5.2 MB/s 2024-03-29T11:08:28.539 DEBUG:teuthology.orchestra.run.smithi018:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:08:28.595 DEBUG:teuthology.orchestra.run.smithi018:> stat /dev/vg_nvme/lv_4 2024-03-29T11:08:28.650 INFO:teuthology.orchestra.run.smithi018.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:08:28.650 INFO:teuthology.orchestra.run.smithi018.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:08:28.651 INFO:teuthology.orchestra.run.smithi018.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-29T11:08:28.651 INFO:teuthology.orchestra.run.smithi018.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:08:28.651 INFO:teuthology.orchestra.run.smithi018.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:08:28.651 INFO:teuthology.orchestra.run.smithi018.stdout:Access: 2024-03-29 11:08:23.217818352 +0000 2024-03-29T11:08:28.651 INFO:teuthology.orchestra.run.smithi018.stdout:Modify: 2024-03-29 11:05:49.461461100 +0000 2024-03-29T11:08:28.651 INFO:teuthology.orchestra.run.smithi018.stdout:Change: 2024-03-29 11:05:49.461461100 +0000 2024-03-29T11:08:28.651 INFO:teuthology.orchestra.run.smithi018.stdout: Birth: 2024-03-29 11:05:49.461461100 +0000 2024-03-29T11:08:28.651 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:08:28.715 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records in 2024-03-29T11:08:28.715 INFO:teuthology.orchestra.run.smithi018.stderr:1+0 records out 2024-03-29T11:08:28.715 INFO:teuthology.orchestra.run.smithi018.stderr:512 bytes copied, 8.1503e-05 s, 6.3 MB/s 2024-03-29T11:08:28.716 DEBUG:teuthology.orchestra.run.smithi018:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:08:28.771 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:08:28.772 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-03-29T11:08:28.772 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:08:28.791 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:08:28.791 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_1 2024-03-29T11:08:28.849 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:08:28.850 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:08:28.850 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-29T11:08:28.850 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:08:28.850 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:08:28.850 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-03-29 11:08:18.871327435 +0000 2024-03-29T11:08:28.850 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-03-29 11:05:44.117491913 +0000 2024-03-29T11:08:28.850 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-03-29 11:05:44.117491913 +0000 2024-03-29T11:08:28.850 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-03-29 11:05:44.117491913 +0000 2024-03-29T11:08:28.850 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:08:28.918 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-03-29T11:08:28.918 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-03-29T11:08:28.918 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000123613 s, 4.1 MB/s 2024-03-29T11:08:28.920 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:08:28.979 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_2 2024-03-29T11:08:29.037 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:08:29.037 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:08:29.037 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-29T11:08:29.037 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:08:29.037 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:08:29.037 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-03-29 11:08:18.871327435 +0000 2024-03-29T11:08:29.037 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-03-29 11:05:44.452481443 +0000 2024-03-29T11:08:29.037 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-03-29 11:05:44.452481443 +0000 2024-03-29T11:08:29.037 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-03-29 11:05:44.452481443 +0000 2024-03-29T11:08:29.037 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:08:29.107 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-03-29T11:08:29.107 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-03-29T11:08:29.107 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000116593 s, 4.4 MB/s 2024-03-29T11:08:29.108 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:08:29.165 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_3 2024-03-29T11:08:29.222 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:08:29.222 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:08:29.223 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-29T11:08:29.223 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:08:29.223 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:08:29.223 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-03-29 11:08:18.871327435 +0000 2024-03-29T11:08:29.223 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-03-29 11:05:44.772471442 +0000 2024-03-29T11:08:29.223 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-03-29 11:05:44.772471442 +0000 2024-03-29T11:08:29.223 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-03-29 11:05:44.772471442 +0000 2024-03-29T11:08:29.223 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:08:29.290 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-03-29T11:08:29.291 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-03-29T11:08:29.291 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000107016 s, 4.8 MB/s 2024-03-29T11:08:29.292 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:08:29.349 DEBUG:teuthology.orchestra.run.smithi070:> stat /dev/vg_nvme/lv_4 2024-03-29T11:08:29.405 INFO:teuthology.orchestra.run.smithi070.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:08:29.405 INFO:teuthology.orchestra.run.smithi070.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:08:29.405 INFO:teuthology.orchestra.run.smithi070.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-29T11:08:29.406 INFO:teuthology.orchestra.run.smithi070.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:08:29.406 INFO:teuthology.orchestra.run.smithi070.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:08:29.406 INFO:teuthology.orchestra.run.smithi070.stdout:Access: 2024-03-29 11:08:18.872327404 +0000 2024-03-29T11:08:29.406 INFO:teuthology.orchestra.run.smithi070.stdout:Modify: 2024-03-29 11:05:45.094461378 +0000 2024-03-29T11:08:29.406 INFO:teuthology.orchestra.run.smithi070.stdout:Change: 2024-03-29 11:05:45.094461378 +0000 2024-03-29T11:08:29.406 INFO:teuthology.orchestra.run.smithi070.stdout: Birth: 2024-03-29 11:05:45.094461378 +0000 2024-03-29T11:08:29.406 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:08:29.475 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records in 2024-03-29T11:08:29.475 INFO:teuthology.orchestra.run.smithi070.stderr:1+0 records out 2024-03-29T11:08:29.475 INFO:teuthology.orchestra.run.smithi070.stderr:512 bytes copied, 0.000110232 s, 4.6 MB/s 2024-03-29T11:08:29.475 DEBUG:teuthology.orchestra.run.smithi070:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:08:29.531 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:08:29.531 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi018.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi070.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:08:29.531 INFO:tasks.ceph:Generating config... 2024-03-29T11:08:29.532 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T11:08:29.532 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T11:08:29.532 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T11:08:29.533 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T11:08:29.533 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T11:08:29.533 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T11:08:29.533 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T11:08:29.533 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T11:08:29.533 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T11:08:29.533 INFO:tasks.ceph:[global] ms type = async 2024-03-29T11:08:29.533 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T11:08:29.533 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T11:08:29.533 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T11:08:29.533 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T11:08:29.533 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T11:08:29.533 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T11:08:29.533 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T11:08:29.533 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T11:08:29.534 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T11:08:29.534 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T11:08:29.534 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T11:08:29.534 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T11:08:29.534 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T11:08:29.534 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-03-29T11:08:29.534 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T11:08:29.534 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T11:08:29.534 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T11:08:29.534 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T11:08:29.534 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T11:08:29.534 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T11:08:29.534 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T11:08:29.534 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T11:08:29.535 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T11:08:29.535 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T11:08:29.535 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T11:08:29.535 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T11:08:29.535 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T11:08:29.535 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T11:08:29.535 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T11:08:29.535 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T11:08:29.535 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T11:08:29.535 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T11:08:29.535 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T11:08:29.535 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T11:08:29.535 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T11:08:29.535 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T11:08:29.535 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T11:08:29.536 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T11:08:29.536 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T11:08:29.536 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T11:08:29.587 INFO:teuthology.orchestra.run.smithi018.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T11:08:29.589 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T11:08:29.635 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:08:29.661 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.18:6789]'), ('mon.c', '[v1:172.21.15.18:6790]'), ('mon.b', '[v1:172.21.15.70:6789]')] 2024-03-29T11:08:29.662 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.18:6789],[v1:172.21.15.18:6790],[v1:172.21.15.70:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T11:08:29.663 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-29T11:08:29.663 DEBUG:teuthology.orchestra.run.smithi018:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:08:29.717 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.18:6789]' --addv c '[v1:172.21.15.18:6790]' --addv b '[v1:172.21.15.70:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:08:29.802 INFO:teuthology.orchestra.run.smithi018.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:08:29.802 INFO:teuthology.orchestra.run.smithi018.stdout:monmaptool: generated fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:08:29.802 INFO:teuthology.orchestra.run.smithi018.stdout:setting min_mon_release = quincy 2024-03-29T11:08:29.802 INFO:teuthology.orchestra.run.smithi018.stdout:epoch 0 2024-03-29T11:08:29.802 INFO:teuthology.orchestra.run.smithi018.stdout:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:08:29.802 INFO:teuthology.orchestra.run.smithi018.stdout:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:08:29.802 INFO:teuthology.orchestra.run.smithi018.stdout:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:08:29.802 INFO:teuthology.orchestra.run.smithi018.stdout:min_mon_release 17 (quincy) 2024-03-29T11:08:29.802 INFO:teuthology.orchestra.run.smithi018.stdout:election_strategy: 1 2024-03-29T11:08:29.802 INFO:teuthology.orchestra.run.smithi018.stdout:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:08:29.802 INFO:teuthology.orchestra.run.smithi018.stdout:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:08:29.803 INFO:teuthology.orchestra.run.smithi018.stdout:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:08:29.803 INFO:teuthology.orchestra.run.smithi018.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T11:08:29.804 DEBUG:teuthology.orchestra.run.smithi018:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:08:29.818 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 303809eb-4027-4a6a-aa53-d48fafbbf1ea... 2024-03-29T11:08:29.819 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:08:29.860 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:08:29.908 INFO:teuthology.orchestra.run.smithi018.stdout:[global] 2024-03-29T11:08:29.908 INFO:teuthology.orchestra.run.smithi018.stdout: chdir = "" 2024-03-29T11:08:29.908 INFO:teuthology.orchestra.run.smithi018.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:08:29.908 INFO:teuthology.orchestra.run.smithi018.stdout: auth supported = cephx 2024-03-29T11:08:29.908 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:29.908 INFO:teuthology.orchestra.run.smithi018.stdout: filestore xattr use omap = true 2024-03-29T11:08:29.908 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:29.909 INFO:teuthology.orchestra.run.smithi018.stdout: mon clock drift allowed = 1.000 2024-03-29T11:08:29.909 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:29.909 INFO:teuthology.orchestra.run.smithi018.stdout: osd crush chooseleaf type = 0 2024-03-29T11:08:29.909 INFO:teuthology.orchestra.run.smithi018.stdout: auth debug = true 2024-03-29T11:08:29.909 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:29.909 INFO:teuthology.orchestra.run.smithi018.stdout: ms die on old message = true 2024-03-29T11:08:29.909 INFO:teuthology.orchestra.run.smithi018.stdout: ms die on bug = true 2024-03-29T11:08:29.909 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:29.909 INFO:teuthology.orchestra.run.smithi018.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:08:29.909 INFO:teuthology.orchestra.run.smithi018.stdout: mon pg warn max object skew = 0 2024-03-29T11:08:29.909 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:29.909 INFO:teuthology.orchestra.run.smithi018.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:08:29.909 INFO:teuthology.orchestra.run.smithi018.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:08:29.910 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:29.910 INFO:teuthology.orchestra.run.smithi018.stdout: osd pool default size = 2 2024-03-29T11:08:29.910 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:29.910 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd allow primary affinity = true 2024-03-29T11:08:29.910 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd allow pg remap = true 2024-03-29T11:08:29.910 INFO:teuthology.orchestra.run.smithi018.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:08:29.910 INFO:teuthology.orchestra.run.smithi018.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:08:29.910 INFO:teuthology.orchestra.run.smithi018.stdout: mon warn on no sortbitwise = false 2024-03-29T11:08:29.910 INFO:teuthology.orchestra.run.smithi018.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:08:29.910 INFO:teuthology.orchestra.run.smithi018.stdout: mon warn on too few osds = false 2024-03-29T11:08:29.910 INFO:teuthology.orchestra.run.smithi018.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:08:29.910 INFO:teuthology.orchestra.run.smithi018.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:08:29.910 INFO:teuthology.orchestra.run.smithi018.stdout: mon_allow_pool_size_one = true 2024-03-29T11:08:29.911 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:29.911 INFO:teuthology.orchestra.run.smithi018.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:08:29.911 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:29.911 INFO:teuthology.orchestra.run.smithi018.stdout: osd default data pool replay window = 5 2024-03-29T11:08:29.911 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:29.911 INFO:teuthology.orchestra.run.smithi018.stdout: mon allow pool delete = true 2024-03-29T11:08:29.911 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:29.911 INFO:teuthology.orchestra.run.smithi018.stdout: mon cluster log file level = debug 2024-03-29T11:08:29.911 INFO:teuthology.orchestra.run.smithi018.stdout: debug asserts on shutdown = true 2024-03-29T11:08:29.911 INFO:teuthology.orchestra.run.smithi018.stdout: mon health detail to clog = false 2024-03-29T11:08:29.911 INFO:teuthology.orchestra.run.smithi018.stdout: mon host = "[v1:172.21.15.18:6789],[v1:172.21.15.18:6790],[v1:172.21.15.70:6789]" 2024-03-29T11:08:29.911 INFO:teuthology.orchestra.run.smithi018.stdout: lockdep = True 2024-03-29T11:08:29.911 INFO:teuthology.orchestra.run.smithi018.stdout: mon client directed command retry = 5 2024-03-29T11:08:29.912 INFO:teuthology.orchestra.run.smithi018.stdout: mon election default strategy = 1 2024-03-29T11:08:29.912 INFO:teuthology.orchestra.run.smithi018.stdout: ms bind msgr2 = False 2024-03-29T11:08:29.912 INFO:teuthology.orchestra.run.smithi018.stdout: ms inject socket failures = 5000 2024-03-29T11:08:29.912 INFO:teuthology.orchestra.run.smithi018.stdout: ms type = async 2024-03-29T11:08:29.912 INFO:teuthology.orchestra.run.smithi018.stdout: fsid = 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:08:29.912 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:29.912 INFO:teuthology.orchestra.run.smithi018.stdout:[osd] 2024-03-29T11:08:29.912 INFO:teuthology.orchestra.run.smithi018.stdout: osd journal size = 100 2024-03-29T11:08:29.912 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:29.912 INFO:teuthology.orchestra.run.smithi018.stdout: osd scrub load threshold = 5.0 2024-03-29T11:08:29.912 INFO:teuthology.orchestra.run.smithi018.stdout: osd scrub max interval = 600 2024-03-29T11:08:29.912 INFO:teuthology.orchestra.run.smithi018.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:08:29.913 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:29.913 INFO:teuthology.orchestra.run.smithi018.stdout: osd recover clone overlap = true 2024-03-29T11:08:29.913 INFO:teuthology.orchestra.run.smithi018.stdout: osd recovery max chunk = 1048576 2024-03-29T11:08:29.913 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:29.913 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug shutdown = true 2024-03-29T11:08:29.913 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug op order = true 2024-03-29T11:08:29.913 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug verify stray on activate = true 2024-03-29T11:08:29.913 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug trim objects = true 2024-03-29T11:08:29.913 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:29.913 INFO:teuthology.orchestra.run.smithi018.stdout: osd open classes on start = true 2024-03-29T11:08:29.913 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug pg log writeout = true 2024-03-29T11:08:29.913 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:29.913 INFO:teuthology.orchestra.run.smithi018.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:08:29.913 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:29.914 INFO:teuthology.orchestra.run.smithi018.stdout: osd map max advance = 10 2024-03-29T11:08:29.914 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:29.914 INFO:teuthology.orchestra.run.smithi018.stdout: journal zero on create = true 2024-03-29T11:08:29.914 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:29.914 INFO:teuthology.orchestra.run.smithi018.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:08:29.914 INFO:teuthology.orchestra.run.smithi018.stdout: filestore apply finisher threads = 3 2024-03-29T11:08:29.914 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:29.914 INFO:teuthology.orchestra.run.smithi018.stdout: bdev debug aio = true 2024-03-29T11:08:29.914 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug misdirected ops = true 2024-03-29T11:08:29.914 INFO:teuthology.orchestra.run.smithi018.stdout: bluestore block size = 96636764160 2024-03-29T11:08:29.914 INFO:teuthology.orchestra.run.smithi018.stdout: bluestore compression algorithm = zstd 2024-03-29T11:08:29.914 INFO:teuthology.orchestra.run.smithi018.stdout: bluestore compression mode = aggressive 2024-03-29T11:08:29.914 INFO:teuthology.orchestra.run.smithi018.stdout: bluestore fsck on mount = True 2024-03-29T11:08:29.914 INFO:teuthology.orchestra.run.smithi018.stdout: bluestore zero block detection = True 2024-03-29T11:08:29.915 INFO:teuthology.orchestra.run.smithi018.stdout: debug bluefs = 20 2024-03-29T11:08:29.915 INFO:teuthology.orchestra.run.smithi018.stdout: debug bluestore = 20 2024-03-29T11:08:29.915 INFO:teuthology.orchestra.run.smithi018.stdout: debug monc = 20 2024-03-29T11:08:29.915 INFO:teuthology.orchestra.run.smithi018.stdout: debug ms = 1 2024-03-29T11:08:29.915 INFO:teuthology.orchestra.run.smithi018.stdout: debug osd = 20 2024-03-29T11:08:29.915 INFO:teuthology.orchestra.run.smithi018.stdout: debug rocksdb = 10 2024-03-29T11:08:29.915 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:08:29.915 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd full ratio = 0.9 2024-03-29T11:08:29.915 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:08:29.915 INFO:teuthology.orchestra.run.smithi018.stdout: osd class default list = * 2024-03-29T11:08:29.915 INFO:teuthology.orchestra.run.smithi018.stdout: osd class load list = * 2024-03-29T11:08:29.915 INFO:teuthology.orchestra.run.smithi018.stdout: osd client watch timeout = 120 2024-03-29T11:08:29.915 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug verify cached snaps = True 2024-03-29T11:08:29.916 INFO:teuthology.orchestra.run.smithi018.stdout: osd debug verify missing on start = True 2024-03-29T11:08:29.916 INFO:teuthology.orchestra.run.smithi018.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:08:29.916 INFO:teuthology.orchestra.run.smithi018.stdout: osd mclock override recovery settings = True 2024-03-29T11:08:29.916 INFO:teuthology.orchestra.run.smithi018.stdout: osd objectstore = bluestore 2024-03-29T11:08:29.916 INFO:teuthology.orchestra.run.smithi018.stdout: osd op queue = debug_random 2024-03-29T11:08:29.916 INFO:teuthology.orchestra.run.smithi018.stdout: osd op queue cut off = debug_random 2024-03-29T11:08:29.916 INFO:teuthology.orchestra.run.smithi018.stdout: osd shutdown pgref assert = True 2024-03-29T11:08:29.916 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:29.916 INFO:teuthology.orchestra.run.smithi018.stdout:[mgr] 2024-03-29T11:08:29.916 INFO:teuthology.orchestra.run.smithi018.stdout: debug ms = 1 2024-03-29T11:08:29.916 INFO:teuthology.orchestra.run.smithi018.stdout: debug mgr = 20 2024-03-29T11:08:29.916 INFO:teuthology.orchestra.run.smithi018.stdout: debug mon = 20 2024-03-29T11:08:29.916 INFO:teuthology.orchestra.run.smithi018.stdout: debug auth = 20 2024-03-29T11:08:29.916 INFO:teuthology.orchestra.run.smithi018.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:08:29.917 INFO:teuthology.orchestra.run.smithi018.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:08:29.917 INFO:teuthology.orchestra.run.smithi018.stdout: mgr/telemetry/nag = false 2024-03-29T11:08:29.917 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:29.917 INFO:teuthology.orchestra.run.smithi018.stdout:[mon] 2024-03-29T11:08:29.917 INFO:teuthology.orchestra.run.smithi018.stdout: debug ms = 1 2024-03-29T11:08:29.917 INFO:teuthology.orchestra.run.smithi018.stdout: debug mon = 20 2024-03-29T11:08:29.917 INFO:teuthology.orchestra.run.smithi018.stdout: debug paxos = 20 2024-03-29T11:08:29.917 INFO:teuthology.orchestra.run.smithi018.stdout: debug auth = 20 2024-03-29T11:08:29.917 INFO:teuthology.orchestra.run.smithi018.stdout: mon data avail warn = 5 2024-03-29T11:08:29.917 INFO:teuthology.orchestra.run.smithi018.stdout: mon mgr mkfs grace = 240 2024-03-29T11:08:29.917 INFO:teuthology.orchestra.run.smithi018.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:08:29.917 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd reporter subtree level = osd 2024-03-29T11:08:29.917 INFO:teuthology.orchestra.run.smithi018.stdout: mon osd prime pg temp = true 2024-03-29T11:08:29.917 INFO:teuthology.orchestra.run.smithi018.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:08:29.918 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:29.918 INFO:teuthology.orchestra.run.smithi018.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:08:29.918 INFO:teuthology.orchestra.run.smithi018.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:08:29.918 INFO:teuthology.orchestra.run.smithi018.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:08:29.918 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:29.918 INFO:teuthology.orchestra.run.smithi018.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:08:29.918 INFO:teuthology.orchestra.run.smithi018.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:08:29.918 INFO:teuthology.orchestra.run.smithi018.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:08:29.918 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:29.918 INFO:teuthology.orchestra.run.smithi018.stdout: # 1m isn't quite enough 2024-03-29T11:08:29.918 INFO:teuthology.orchestra.run.smithi018.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:08:29.918 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:29.918 INFO:teuthology.orchestra.run.smithi018.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:08:29.919 INFO:teuthology.orchestra.run.smithi018.stdout: mon min osdmap epochs = 50 2024-03-29T11:08:29.919 INFO:teuthology.orchestra.run.smithi018.stdout: mon osdmap full prune interval = 2 2024-03-29T11:08:29.919 INFO:teuthology.orchestra.run.smithi018.stdout: mon osdmap full prune min = 15 2024-03-29T11:08:29.919 INFO:teuthology.orchestra.run.smithi018.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:08:29.919 INFO:teuthology.orchestra.run.smithi018.stdout: mon scrub interval = 300 2024-03-29T11:08:29.919 INFO:teuthology.orchestra.run.smithi018.stdout: mon warn on pool no app = False 2024-03-29T11:08:29.919 INFO:teuthology.orchestra.run.smithi018.stdout: paxos service trim min = 10 2024-03-29T11:08:29.919 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:08:29.919 INFO:teuthology.orchestra.run.smithi018.stdout:[client] 2024-03-29T11:08:29.919 INFO:teuthology.orchestra.run.smithi018.stdout: rgw cache enabled = true 2024-03-29T11:08:29.919 INFO:teuthology.orchestra.run.smithi018.stdout: rgw enable ops log = true 2024-03-29T11:08:29.919 INFO:teuthology.orchestra.run.smithi018.stdout: rgw enable usage log = true 2024-03-29T11:08:29.919 INFO:teuthology.orchestra.run.smithi018.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:08:29.919 INFO:teuthology.orchestra.run.smithi018.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:08:29.920 INFO:teuthology.orchestra.run.smithi018.stdout: debug monc = 20 2024-03-29T11:08:29.920 INFO:teuthology.orchestra.run.smithi018.stdout: debug ms = 1 2024-03-29T11:08:29.920 INFO:teuthology.orchestra.run.smithi018.stdout: debug objecter = 20 2024-03-29T11:08:29.920 INFO:teuthology.orchestra.run.smithi018.stdout: debug rados = 20 2024-03-29T11:08:29.920 INFO:teuthology.orchestra.run.smithi018.stdout:[mon.a] 2024-03-29T11:08:29.920 INFO:teuthology.orchestra.run.smithi018.stdout:[mon.c] 2024-03-29T11:08:29.920 INFO:teuthology.orchestra.run.smithi018.stdout:[mon.b] 2024-03-29T11:08:29.920 INFO:teuthology.orchestra.run.smithi070.stdout:[global] 2024-03-29T11:08:29.921 INFO:teuthology.orchestra.run.smithi070.stdout: chdir = "" 2024-03-29T11:08:29.921 INFO:teuthology.orchestra.run.smithi070.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:08:29.921 INFO:teuthology.orchestra.run.smithi070.stdout: auth supported = cephx 2024-03-29T11:08:29.921 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:29.921 INFO:teuthology.orchestra.run.smithi070.stdout: filestore xattr use omap = true 2024-03-29T11:08:29.921 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:29.921 INFO:teuthology.orchestra.run.smithi070.stdout: mon clock drift allowed = 1.000 2024-03-29T11:08:29.921 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:29.921 INFO:teuthology.orchestra.run.smithi070.stdout: osd crush chooseleaf type = 0 2024-03-29T11:08:29.921 INFO:teuthology.orchestra.run.smithi070.stdout: auth debug = true 2024-03-29T11:08:29.921 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:29.921 INFO:teuthology.orchestra.run.smithi070.stdout: ms die on old message = true 2024-03-29T11:08:29.922 INFO:teuthology.orchestra.run.smithi070.stdout: ms die on bug = true 2024-03-29T11:08:29.922 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:29.922 INFO:teuthology.orchestra.run.smithi070.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:08:29.922 INFO:teuthology.orchestra.run.smithi070.stdout: mon pg warn max object skew = 0 2024-03-29T11:08:29.922 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:29.922 INFO:teuthology.orchestra.run.smithi070.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:08:29.922 INFO:teuthology.orchestra.run.smithi070.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:08:29.922 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:29.922 INFO:teuthology.orchestra.run.smithi070.stdout: osd pool default size = 2 2024-03-29T11:08:29.922 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:29.922 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd allow primary affinity = true 2024-03-29T11:08:29.922 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd allow pg remap = true 2024-03-29T11:08:29.922 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:08:29.922 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:08:29.923 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on no sortbitwise = false 2024-03-29T11:08:29.923 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:08:29.923 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on too few osds = false 2024-03-29T11:08:29.923 INFO:teuthology.orchestra.run.smithi070.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:08:29.923 INFO:teuthology.orchestra.run.smithi070.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:08:29.923 INFO:teuthology.orchestra.run.smithi070.stdout: mon_allow_pool_size_one = true 2024-03-29T11:08:29.923 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:29.923 INFO:teuthology.orchestra.run.smithi070.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:08:29.923 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:29.923 INFO:teuthology.orchestra.run.smithi070.stdout: osd default data pool replay window = 5 2024-03-29T11:08:29.923 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:29.923 INFO:teuthology.orchestra.run.smithi070.stdout: mon allow pool delete = true 2024-03-29T11:08:29.923 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:29.923 INFO:teuthology.orchestra.run.smithi070.stdout: mon cluster log file level = debug 2024-03-29T11:08:29.924 INFO:teuthology.orchestra.run.smithi070.stdout: debug asserts on shutdown = true 2024-03-29T11:08:29.924 INFO:teuthology.orchestra.run.smithi070.stdout: mon health detail to clog = false 2024-03-29T11:08:29.924 INFO:teuthology.orchestra.run.smithi070.stdout: mon host = "[v1:172.21.15.18:6789],[v1:172.21.15.18:6790],[v1:172.21.15.70:6789]" 2024-03-29T11:08:29.924 INFO:teuthology.orchestra.run.smithi070.stdout: lockdep = True 2024-03-29T11:08:29.924 INFO:teuthology.orchestra.run.smithi070.stdout: mon client directed command retry = 5 2024-03-29T11:08:29.924 INFO:teuthology.orchestra.run.smithi070.stdout: mon election default strategy = 1 2024-03-29T11:08:29.924 INFO:teuthology.orchestra.run.smithi070.stdout: ms bind msgr2 = False 2024-03-29T11:08:29.924 INFO:teuthology.orchestra.run.smithi070.stdout: ms inject socket failures = 5000 2024-03-29T11:08:29.924 INFO:teuthology.orchestra.run.smithi070.stdout: ms type = async 2024-03-29T11:08:29.924 INFO:teuthology.orchestra.run.smithi070.stdout: fsid = 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:08:29.924 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:29.924 INFO:teuthology.orchestra.run.smithi070.stdout:[osd] 2024-03-29T11:08:29.924 INFO:teuthology.orchestra.run.smithi070.stdout: osd journal size = 100 2024-03-29T11:08:29.924 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:29.924 INFO:teuthology.orchestra.run.smithi070.stdout: osd scrub load threshold = 5.0 2024-03-29T11:08:29.925 INFO:teuthology.orchestra.run.smithi070.stdout: osd scrub max interval = 600 2024-03-29T11:08:29.925 INFO:teuthology.orchestra.run.smithi070.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:08:29.925 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:29.925 INFO:teuthology.orchestra.run.smithi070.stdout: osd recover clone overlap = true 2024-03-29T11:08:29.925 INFO:teuthology.orchestra.run.smithi070.stdout: osd recovery max chunk = 1048576 2024-03-29T11:08:29.925 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:29.925 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug shutdown = true 2024-03-29T11:08:29.925 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug op order = true 2024-03-29T11:08:29.925 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug verify stray on activate = true 2024-03-29T11:08:29.925 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug trim objects = true 2024-03-29T11:08:29.925 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:29.925 INFO:teuthology.orchestra.run.smithi070.stdout: osd open classes on start = true 2024-03-29T11:08:29.925 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug pg log writeout = true 2024-03-29T11:08:29.925 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:29.925 INFO:teuthology.orchestra.run.smithi070.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:08:29.926 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:29.926 INFO:teuthology.orchestra.run.smithi070.stdout: osd map max advance = 10 2024-03-29T11:08:29.926 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:29.926 INFO:teuthology.orchestra.run.smithi070.stdout: journal zero on create = true 2024-03-29T11:08:29.926 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:29.926 INFO:teuthology.orchestra.run.smithi070.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:08:29.926 INFO:teuthology.orchestra.run.smithi070.stdout: filestore apply finisher threads = 3 2024-03-29T11:08:29.926 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:29.926 INFO:teuthology.orchestra.run.smithi070.stdout: bdev debug aio = true 2024-03-29T11:08:29.926 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug misdirected ops = true 2024-03-29T11:08:29.926 INFO:teuthology.orchestra.run.smithi070.stdout: bluestore block size = 96636764160 2024-03-29T11:08:29.926 INFO:teuthology.orchestra.run.smithi070.stdout: bluestore compression algorithm = zstd 2024-03-29T11:08:29.926 INFO:teuthology.orchestra.run.smithi070.stdout: bluestore compression mode = aggressive 2024-03-29T11:08:29.926 INFO:teuthology.orchestra.run.smithi070.stdout: bluestore fsck on mount = True 2024-03-29T11:08:29.927 INFO:teuthology.orchestra.run.smithi070.stdout: bluestore zero block detection = True 2024-03-29T11:08:29.927 INFO:teuthology.orchestra.run.smithi070.stdout: debug bluefs = 20 2024-03-29T11:08:29.927 INFO:teuthology.orchestra.run.smithi070.stdout: debug bluestore = 20 2024-03-29T11:08:29.927 INFO:teuthology.orchestra.run.smithi070.stdout: debug monc = 20 2024-03-29T11:08:29.927 INFO:teuthology.orchestra.run.smithi070.stdout: debug ms = 1 2024-03-29T11:08:29.927 INFO:teuthology.orchestra.run.smithi070.stdout: debug osd = 20 2024-03-29T11:08:29.927 INFO:teuthology.orchestra.run.smithi070.stdout: debug rocksdb = 10 2024-03-29T11:08:29.927 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:08:29.927 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd full ratio = 0.9 2024-03-29T11:08:29.927 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:08:29.927 INFO:teuthology.orchestra.run.smithi070.stdout: osd class default list = * 2024-03-29T11:08:29.927 INFO:teuthology.orchestra.run.smithi070.stdout: osd class load list = * 2024-03-29T11:08:29.927 INFO:teuthology.orchestra.run.smithi070.stdout: osd client watch timeout = 120 2024-03-29T11:08:29.927 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug verify cached snaps = True 2024-03-29T11:08:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: osd debug verify missing on start = True 2024-03-29T11:08:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:08:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: osd mclock override recovery settings = True 2024-03-29T11:08:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: osd objectstore = bluestore 2024-03-29T11:08:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: osd op queue = debug_random 2024-03-29T11:08:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: osd op queue cut off = debug_random 2024-03-29T11:08:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: osd shutdown pgref assert = True 2024-03-29T11:08:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:29.928 INFO:teuthology.orchestra.run.smithi070.stdout:[mgr] 2024-03-29T11:08:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: debug ms = 1 2024-03-29T11:08:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: debug mgr = 20 2024-03-29T11:08:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: debug mon = 20 2024-03-29T11:08:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: debug auth = 20 2024-03-29T11:08:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:08:29.928 INFO:teuthology.orchestra.run.smithi070.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:08:29.929 INFO:teuthology.orchestra.run.smithi070.stdout: mgr/telemetry/nag = false 2024-03-29T11:08:29.929 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:29.929 INFO:teuthology.orchestra.run.smithi070.stdout:[mon] 2024-03-29T11:08:29.929 INFO:teuthology.orchestra.run.smithi070.stdout: debug ms = 1 2024-03-29T11:08:29.929 INFO:teuthology.orchestra.run.smithi070.stdout: debug mon = 20 2024-03-29T11:08:29.929 INFO:teuthology.orchestra.run.smithi070.stdout: debug paxos = 20 2024-03-29T11:08:29.929 INFO:teuthology.orchestra.run.smithi070.stdout: debug auth = 20 2024-03-29T11:08:29.929 INFO:teuthology.orchestra.run.smithi070.stdout: mon data avail warn = 5 2024-03-29T11:08:29.929 INFO:teuthology.orchestra.run.smithi070.stdout: mon mgr mkfs grace = 240 2024-03-29T11:08:29.929 INFO:teuthology.orchestra.run.smithi070.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:08:29.929 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd reporter subtree level = osd 2024-03-29T11:08:29.929 INFO:teuthology.orchestra.run.smithi070.stdout: mon osd prime pg temp = true 2024-03-29T11:08:29.929 INFO:teuthology.orchestra.run.smithi070.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:08:29.929 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:29.930 INFO:teuthology.orchestra.run.smithi070.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:08:29.930 INFO:teuthology.orchestra.run.smithi070.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:08:29.930 INFO:teuthology.orchestra.run.smithi070.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:08:29.930 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:29.930 INFO:teuthology.orchestra.run.smithi070.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:08:29.930 INFO:teuthology.orchestra.run.smithi070.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:08:29.930 INFO:teuthology.orchestra.run.smithi070.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:08:29.930 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:29.930 INFO:teuthology.orchestra.run.smithi070.stdout: # 1m isn't quite enough 2024-03-29T11:08:29.930 INFO:teuthology.orchestra.run.smithi070.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:08:29.930 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:29.930 INFO:teuthology.orchestra.run.smithi070.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:08:29.930 INFO:teuthology.orchestra.run.smithi070.stdout: mon min osdmap epochs = 50 2024-03-29T11:08:29.930 INFO:teuthology.orchestra.run.smithi070.stdout: mon osdmap full prune interval = 2 2024-03-29T11:08:29.930 INFO:teuthology.orchestra.run.smithi070.stdout: mon osdmap full prune min = 15 2024-03-29T11:08:29.931 INFO:teuthology.orchestra.run.smithi070.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:08:29.931 INFO:teuthology.orchestra.run.smithi070.stdout: mon scrub interval = 300 2024-03-29T11:08:29.931 INFO:teuthology.orchestra.run.smithi070.stdout: mon warn on pool no app = False 2024-03-29T11:08:29.931 INFO:teuthology.orchestra.run.smithi070.stdout: paxos service trim min = 10 2024-03-29T11:08:29.931 INFO:teuthology.orchestra.run.smithi070.stdout: 2024-03-29T11:08:29.931 INFO:teuthology.orchestra.run.smithi070.stdout:[client] 2024-03-29T11:08:29.931 INFO:teuthology.orchestra.run.smithi070.stdout: rgw cache enabled = true 2024-03-29T11:08:29.931 INFO:teuthology.orchestra.run.smithi070.stdout: rgw enable ops log = true 2024-03-29T11:08:29.931 INFO:teuthology.orchestra.run.smithi070.stdout: rgw enable usage log = true 2024-03-29T11:08:29.931 INFO:teuthology.orchestra.run.smithi070.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:08:29.931 INFO:teuthology.orchestra.run.smithi070.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:08:29.931 INFO:teuthology.orchestra.run.smithi070.stdout: debug monc = 20 2024-03-29T11:08:29.931 INFO:teuthology.orchestra.run.smithi070.stdout: debug ms = 1 2024-03-29T11:08:29.931 INFO:teuthology.orchestra.run.smithi070.stdout: debug objecter = 20 2024-03-29T11:08:29.931 INFO:teuthology.orchestra.run.smithi070.stdout: debug rados = 20 2024-03-29T11:08:29.932 INFO:teuthology.orchestra.run.smithi070.stdout:[mon.a] 2024-03-29T11:08:29.932 INFO:teuthology.orchestra.run.smithi070.stdout:[mon.c] 2024-03-29T11:08:29.932 INFO:teuthology.orchestra.run.smithi070.stdout:[mon.b] 2024-03-29T11:08:29.933 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T11:08:29.933 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T11:08:30.035 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T11:08:30.035 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-29T11:08:30.035 DEBUG:teuthology.orchestra.run.smithi018:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T11:08:30.050 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-29T11:08:30.050 DEBUG:teuthology.orchestra.run.smithi018:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T11:08:30.105 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi018.front.sepia.ceph.com 2024-03-29T11:08:30.105 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-29T11:08:30.105 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:08:30.106 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:08:30.184 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-29T11:08:30.184 DEBUG:teuthology.orchestra.run.smithi018:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:08:30.240 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi070.front.sepia.ceph.com 2024-03-29T11:08:30.242 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-03-29T11:08:30.242 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:08:30.242 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:08:30.285 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-03-29T11:08:30.285 DEBUG:teuthology.orchestra.run.smithi070:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:08:30.341 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T11:08:30.341 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T11:08:30.342 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:08:30.395 INFO:teuthology.orchestra.run.smithi018.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:08:30.398 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:08:30.464 INFO:teuthology.orchestra.run.smithi070.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:08:30.466 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T11:08:30.466 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T11:08:30.467 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T11:08:30.508 INFO:teuthology.orchestra.run.smithi018.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T11:08:30.523 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T11:08:30.566 INFO:teuthology.orchestra.run.smithi070.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T11:08:30.582 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T11:08:30.582 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi018.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi070.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:08:30.582 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T11:08:30.608 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:08:30.608 INFO:tasks.ceph:role: osd.0 2024-03-29T11:08:30.608 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi018.front.sepia.ceph.com 2024-03-29T11:08:30.608 DEBUG:teuthology.orchestra.run.smithi018:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:08:30.674 INFO:teuthology.orchestra.run.smithi018.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:08:30.675 INFO:teuthology.orchestra.run.smithi018.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:08:30.675 INFO:teuthology.orchestra.run.smithi018.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:08:30.675 INFO:teuthology.orchestra.run.smithi018.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:08:30.675 INFO:teuthology.orchestra.run.smithi018.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:08:30.675 INFO:teuthology.orchestra.run.smithi018.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:08:30.675 INFO:teuthology.orchestra.run.smithi018.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:08:30.676 INFO:teuthology.orchestra.run.smithi018.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:08:30.676 INFO:teuthology.orchestra.run.smithi018.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:08:30.676 INFO:teuthology.orchestra.run.smithi018.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:08:30.706 INFO:teuthology.orchestra.run.smithi018.stdout:Discarding blocks...Done. 2024-03-29T11:08:30.708 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi018.front.sepia.ceph.com -o noatime 2024-03-29T11:08:30.708 DEBUG:teuthology.orchestra.run.smithi018:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T11:08:30.779 DEBUG:teuthology.orchestra.run.smithi018:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T11:08:30.847 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T11:08:30.914 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:08:30.914 INFO:tasks.ceph:role: osd.1 2024-03-29T11:08:30.914 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi018.front.sepia.ceph.com 2024-03-29T11:08:30.914 DEBUG:teuthology.orchestra.run.smithi018:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:08:30.980 INFO:teuthology.orchestra.run.smithi018.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:08:30.980 INFO:teuthology.orchestra.run.smithi018.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:08:30.980 INFO:teuthology.orchestra.run.smithi018.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:08:30.980 INFO:teuthology.orchestra.run.smithi018.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:08:30.980 INFO:teuthology.orchestra.run.smithi018.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:08:30.980 INFO:teuthology.orchestra.run.smithi018.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:08:30.980 INFO:teuthology.orchestra.run.smithi018.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:08:30.981 INFO:teuthology.orchestra.run.smithi018.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:08:30.981 INFO:teuthology.orchestra.run.smithi018.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:08:30.981 INFO:teuthology.orchestra.run.smithi018.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:08:31.012 INFO:teuthology.orchestra.run.smithi018.stdout:Discarding blocks...Done. 2024-03-29T11:08:31.014 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi018.front.sepia.ceph.com -o noatime 2024-03-29T11:08:31.014 DEBUG:teuthology.orchestra.run.smithi018:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T11:08:31.085 DEBUG:teuthology.orchestra.run.smithi018:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T11:08:31.156 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T11:08:31.223 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:08:31.223 INFO:tasks.ceph:role: osd.2 2024-03-29T11:08:31.223 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi018.front.sepia.ceph.com 2024-03-29T11:08:31.224 DEBUG:teuthology.orchestra.run.smithi018:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:08:31.289 INFO:teuthology.orchestra.run.smithi018.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:08:31.290 INFO:teuthology.orchestra.run.smithi018.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:08:31.290 INFO:teuthology.orchestra.run.smithi018.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:08:31.290 INFO:teuthology.orchestra.run.smithi018.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:08:31.290 INFO:teuthology.orchestra.run.smithi018.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:08:31.290 INFO:teuthology.orchestra.run.smithi018.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:08:31.290 INFO:teuthology.orchestra.run.smithi018.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:08:31.290 INFO:teuthology.orchestra.run.smithi018.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:08:31.290 INFO:teuthology.orchestra.run.smithi018.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:08:31.290 INFO:teuthology.orchestra.run.smithi018.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:08:31.322 INFO:teuthology.orchestra.run.smithi018.stdout:Discarding blocks...Done. 2024-03-29T11:08:31.324 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi018.front.sepia.ceph.com -o noatime 2024-03-29T11:08:31.324 DEBUG:teuthology.orchestra.run.smithi018:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T11:08:31.395 DEBUG:teuthology.orchestra.run.smithi018:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T11:08:31.467 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T11:08:31.535 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:08:31.535 INFO:tasks.ceph:role: osd.3 2024-03-29T11:08:31.535 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi018.front.sepia.ceph.com 2024-03-29T11:08:31.536 DEBUG:teuthology.orchestra.run.smithi018:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:08:31.602 INFO:teuthology.orchestra.run.smithi018.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:08:31.603 INFO:teuthology.orchestra.run.smithi018.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:08:31.603 INFO:teuthology.orchestra.run.smithi018.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:08:31.603 INFO:teuthology.orchestra.run.smithi018.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:08:31.603 INFO:teuthology.orchestra.run.smithi018.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:08:31.603 INFO:teuthology.orchestra.run.smithi018.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:08:31.603 INFO:teuthology.orchestra.run.smithi018.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:08:31.603 INFO:teuthology.orchestra.run.smithi018.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:08:31.603 INFO:teuthology.orchestra.run.smithi018.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:08:31.603 INFO:teuthology.orchestra.run.smithi018.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:08:31.635 INFO:teuthology.orchestra.run.smithi018.stdout:Discarding blocks...Done. 2024-03-29T11:08:31.637 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi018.front.sepia.ceph.com -o noatime 2024-03-29T11:08:31.637 DEBUG:teuthology.orchestra.run.smithi018:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T11:08:31.708 DEBUG:teuthology.orchestra.run.smithi018:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T11:08:31.779 DEBUG:teuthology.orchestra.run.smithi018:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:08:31.887 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:31.908+0000 7f04ba417640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T11:08:31.888 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:31.909+0000 7f04ba417640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T11:08:31.888 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:31.909+0000 7f04ba417640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:08:31.888 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:31.909+0000 7f04ba417640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:08:31.888 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:31.909+0000 7f04ba417640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:08:31.888 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:31.909+0000 7f04ba417640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T11:08:34.738 DEBUG:teuthology.orchestra.run.smithi018:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T11:08:34.769 DEBUG:teuthology.orchestra.run.smithi018:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:08:34.862 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:34.883+0000 7f289e42a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T11:08:34.862 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:34.883+0000 7f289e42a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T11:08:34.862 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:34.883+0000 7f289e42a640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:08:34.862 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:34.883+0000 7f289e42a640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:08:34.862 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:34.883+0000 7f289e42a640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:08:34.862 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:34.884+0000 7f289e42a640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T11:08:37.704 DEBUG:teuthology.orchestra.run.smithi018:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T11:08:37.735 DEBUG:teuthology.orchestra.run.smithi018:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:08:37.828 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:37.849+0000 7f91d99e1640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T11:08:37.828 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:37.849+0000 7f91d99e1640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T11:08:37.828 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:37.849+0000 7f91d99e1640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:08:37.828 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:37.849+0000 7f91d99e1640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:08:37.828 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:37.849+0000 7f91d99e1640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:08:37.829 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:37.850+0000 7f91d99e1640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T11:08:40.676 DEBUG:teuthology.orchestra.run.smithi018:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T11:08:40.706 DEBUG:teuthology.orchestra.run.smithi018:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:08:40.798 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:40.819+0000 7febd3349640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T11:08:40.798 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:40.819+0000 7febd3349640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T11:08:40.798 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:40.819+0000 7febd3349640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:08:40.798 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:40.819+0000 7febd3349640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:08:40.798 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:40.819+0000 7febd3349640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:08:40.798 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:40.819+0000 7febd3349640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T11:08:43.640 DEBUG:teuthology.orchestra.run.smithi018:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T11:08:43.671 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T11:08:43.703 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:08:43.703 INFO:tasks.ceph:role: osd.4 2024-03-29T11:08:43.703 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi070.front.sepia.ceph.com 2024-03-29T11:08:43.703 DEBUG:teuthology.orchestra.run.smithi070:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:08:43.773 INFO:teuthology.orchestra.run.smithi070.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:08:43.774 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:08:43.774 INFO:teuthology.orchestra.run.smithi070.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:08:43.774 INFO:teuthology.orchestra.run.smithi070.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:08:43.774 INFO:teuthology.orchestra.run.smithi070.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:08:43.774 INFO:teuthology.orchestra.run.smithi070.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:08:43.774 INFO:teuthology.orchestra.run.smithi070.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:08:43.774 INFO:teuthology.orchestra.run.smithi070.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:08:43.774 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:08:43.774 INFO:teuthology.orchestra.run.smithi070.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:08:43.807 INFO:teuthology.orchestra.run.smithi070.stdout:Discarding blocks...Done. 2024-03-29T11:08:43.809 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi070.front.sepia.ceph.com -o noatime 2024-03-29T11:08:43.809 DEBUG:teuthology.orchestra.run.smithi070:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T11:08:43.881 DEBUG:teuthology.orchestra.run.smithi070:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T11:08:43.954 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T11:08:44.022 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:08:44.022 INFO:tasks.ceph:role: osd.5 2024-03-29T11:08:44.022 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi070.front.sepia.ceph.com 2024-03-29T11:08:44.022 DEBUG:teuthology.orchestra.run.smithi070:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:08:44.089 INFO:teuthology.orchestra.run.smithi070.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:08:44.089 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:08:44.089 INFO:teuthology.orchestra.run.smithi070.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:08:44.089 INFO:teuthology.orchestra.run.smithi070.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:08:44.089 INFO:teuthology.orchestra.run.smithi070.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:08:44.089 INFO:teuthology.orchestra.run.smithi070.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:08:44.089 INFO:teuthology.orchestra.run.smithi070.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:08:44.089 INFO:teuthology.orchestra.run.smithi070.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:08:44.089 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:08:44.089 INFO:teuthology.orchestra.run.smithi070.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:08:44.122 INFO:teuthology.orchestra.run.smithi070.stdout:Discarding blocks...Done. 2024-03-29T11:08:44.123 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi070.front.sepia.ceph.com -o noatime 2024-03-29T11:08:44.124 DEBUG:teuthology.orchestra.run.smithi070:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T11:08:44.194 DEBUG:teuthology.orchestra.run.smithi070:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T11:08:44.265 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T11:08:44.333 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:08:44.334 INFO:tasks.ceph:role: osd.6 2024-03-29T11:08:44.334 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi070.front.sepia.ceph.com 2024-03-29T11:08:44.334 DEBUG:teuthology.orchestra.run.smithi070:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:08:44.400 INFO:teuthology.orchestra.run.smithi070.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:08:44.401 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:08:44.401 INFO:teuthology.orchestra.run.smithi070.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:08:44.401 INFO:teuthology.orchestra.run.smithi070.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:08:44.401 INFO:teuthology.orchestra.run.smithi070.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:08:44.401 INFO:teuthology.orchestra.run.smithi070.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:08:44.401 INFO:teuthology.orchestra.run.smithi070.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:08:44.401 INFO:teuthology.orchestra.run.smithi070.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:08:44.401 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:08:44.401 INFO:teuthology.orchestra.run.smithi070.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:08:44.433 INFO:teuthology.orchestra.run.smithi070.stdout:Discarding blocks...Done. 2024-03-29T11:08:44.435 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi070.front.sepia.ceph.com -o noatime 2024-03-29T11:08:44.436 DEBUG:teuthology.orchestra.run.smithi070:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T11:08:44.511 DEBUG:teuthology.orchestra.run.smithi070:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T11:08:44.582 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T11:08:44.649 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:08:44.650 INFO:tasks.ceph:role: osd.7 2024-03-29T11:08:44.650 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi070.front.sepia.ceph.com 2024-03-29T11:08:44.650 DEBUG:teuthology.orchestra.run.smithi070:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:08:44.715 INFO:teuthology.orchestra.run.smithi070.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:08:44.715 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:08:44.716 INFO:teuthology.orchestra.run.smithi070.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:08:44.716 INFO:teuthology.orchestra.run.smithi070.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:08:44.716 INFO:teuthology.orchestra.run.smithi070.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:08:44.716 INFO:teuthology.orchestra.run.smithi070.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:08:44.716 INFO:teuthology.orchestra.run.smithi070.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:08:44.716 INFO:teuthology.orchestra.run.smithi070.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:08:44.716 INFO:teuthology.orchestra.run.smithi070.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:08:44.716 INFO:teuthology.orchestra.run.smithi070.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:08:44.748 INFO:teuthology.orchestra.run.smithi070.stdout:Discarding blocks...Done. 2024-03-29T11:08:44.750 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi070.front.sepia.ceph.com -o noatime 2024-03-29T11:08:44.751 DEBUG:teuthology.orchestra.run.smithi070:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T11:08:44.820 DEBUG:teuthology.orchestra.run.smithi070:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T11:08:44.890 DEBUG:teuthology.orchestra.run.smithi070:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:08:44.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:08:45.001+0000 7f00391ac640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T11:08:44.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:08:45.002+0000 7f00391ac640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T11:08:44.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:08:45.002+0000 7f00391ac640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:08:44.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:08:45.002+0000 7f00391ac640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:08:44.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:08:45.002+0000 7f00391ac640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:08:44.981 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:08:45.002+0000 7f00391ac640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T11:08:47.822 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T11:08:47.852 DEBUG:teuthology.orchestra.run.smithi070:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:08:47.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:08:47.961+0000 7f38c6a10640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T11:08:47.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:08:47.961+0000 7f38c6a10640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T11:08:47.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:08:47.961+0000 7f38c6a10640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:08:47.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:08:47.961+0000 7f38c6a10640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:08:47.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:08:47.961+0000 7f38c6a10640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:08:47.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:08:47.962+0000 7f38c6a10640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T11:08:50.789 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T11:08:50.820 DEBUG:teuthology.orchestra.run.smithi070:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:08:50.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:08:50.933+0000 7f3abca0d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T11:08:50.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:08:50.933+0000 7f3abca0d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T11:08:50.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:08:50.933+0000 7f3abca0d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:08:50.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:08:50.933+0000 7f3abca0d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:08:50.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:08:50.933+0000 7f3abca0d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:08:50.912 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:08:50.933+0000 7f3abca0d640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T11:08:53.755 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T11:08:53.786 DEBUG:teuthology.orchestra.run.smithi070:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:08:53.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:08:53.898+0000 7ff854e24640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T11:08:53.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:08:53.898+0000 7ff854e24640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T11:08:53.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:08:53.898+0000 7ff854e24640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:08:53.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:08:53.898+0000 7ff854e24640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:08:53.877 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:08:53.898+0000 7ff854e24640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:08:53.878 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:08:53.899+0000 7ff854e24640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T11:08:56.724 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T11:08:56.755 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T11:08:56.755 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-29T11:08:56.755 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T11:08:56.787 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-29T11:08:56.788 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T11:08:56.859 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-29T11:08:56.859 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T11:08:56.929 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-29T11:08:56.929 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T11:08:56.998 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-29T11:08:56.998 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T11:08:57.066 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-03-29T11:08:57.067 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T11:08:57.098 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-03-29T11:08:57.098 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T11:08:57.170 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-03-29T11:08:57.170 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T11:08:57.240 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-03-29T11:08:57.240 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T11:08:57.309 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-03-29T11:08:57.309 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T11:08:57.378 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-29T11:08:57.378 DEBUG:teuthology.orchestra.run.smithi018:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T11:08:57.394 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-03-29T11:08:57.395 DEBUG:teuthology.orchestra.run.smithi070:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T11:08:57.433 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T11:08:57.433 DEBUG:teuthology.orchestra.run.smithi018:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:08:57.437 DEBUG:teuthology.orchestra.run.smithi070:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:08:57.476 INFO:teuthology.orchestra.run.smithi018.stdout:[mgr.y] 2024-03-29T11:08:57.476 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQAuoQZm0xjxGBAAjivrnXF+geJMjmLZSeZExA== 2024-03-29T11:08:57.476 INFO:teuthology.orchestra.run.smithi018.stdout:[osd.0] 2024-03-29T11:08:57.476 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQAvoQZmL9s5NhAA4Y3xeW12BVpaQu/Y61S8pQ== 2024-03-29T11:08:57.476 INFO:teuthology.orchestra.run.smithi018.stdout:[osd.1] 2024-03-29T11:08:57.477 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQAyoQZmVR+2NBAAZH82GWPeHm93wcQJOTFaXQ== 2024-03-29T11:08:57.477 INFO:teuthology.orchestra.run.smithi018.stdout:[osd.2] 2024-03-29T11:08:57.477 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQA1oQZmrp+wMhAAUYYcC0oUAQ7cK/GNddzIjQ== 2024-03-29T11:08:57.477 INFO:teuthology.orchestra.run.smithi018.stdout:[osd.3] 2024-03-29T11:08:57.477 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQA4oQZm+K/gMBAAhA8a7qSnQSYdkUoC6wUHhg== 2024-03-29T11:08:57.477 INFO:teuthology.orchestra.run.smithi018.stdout:[mgr.x] 2024-03-29T11:08:57.477 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQAuoQZmsv4AHRAAXnB6t/t/YZed82UeRxiWPg== 2024-03-29T11:08:57.477 INFO:teuthology.orchestra.run.smithi018.stdout:[osd.4] 2024-03-29T11:08:57.477 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQA9oQZmC68tABAAGJOOC06tLIuNULpTKz/gMg== 2024-03-29T11:08:57.477 INFO:teuthology.orchestra.run.smithi018.stdout:[osd.5] 2024-03-29T11:08:57.477 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQA/oQZmm6ReORAAwnGxtr8KaempKsyoIZrALg== 2024-03-29T11:08:57.477 INFO:teuthology.orchestra.run.smithi018.stdout:[osd.6] 2024-03-29T11:08:57.477 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQBCoQZmEUivNxAAxnlWAzYjfcfVScuoI2Z13g== 2024-03-29T11:08:57.478 INFO:teuthology.orchestra.run.smithi018.stdout:[osd.7] 2024-03-29T11:08:57.478 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQBFoQZm8zWdNRAA5N8j5jucRcozgN34STfZ7w== 2024-03-29T11:08:57.478 INFO:teuthology.orchestra.run.smithi018.stdout:[client.0] 2024-03-29T11:08:57.478 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQAuoQZmmLeoHxAAgsMVwY38jg2DDAcy7cbsng== 2024-03-29T11:08:57.478 INFO:teuthology.orchestra.run.smithi018.stdout:[client.1] 2024-03-29T11:08:57.478 INFO:teuthology.orchestra.run.smithi018.stdout: key = AQAuoQZm8KYhIxAARpiJLg7dC0lCRmCd6JIgDA== 2024-03-29T11:08:57.499 INFO:teuthology.orchestra.run.smithi070.stdout:[mgr.y] 2024-03-29T11:08:57.499 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQAuoQZm0xjxGBAAjivrnXF+geJMjmLZSeZExA== 2024-03-29T11:08:57.499 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.0] 2024-03-29T11:08:57.500 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQAvoQZmL9s5NhAA4Y3xeW12BVpaQu/Y61S8pQ== 2024-03-29T11:08:57.500 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.1] 2024-03-29T11:08:57.500 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQAyoQZmVR+2NBAAZH82GWPeHm93wcQJOTFaXQ== 2024-03-29T11:08:57.500 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.2] 2024-03-29T11:08:57.500 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQA1oQZmrp+wMhAAUYYcC0oUAQ7cK/GNddzIjQ== 2024-03-29T11:08:57.500 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.3] 2024-03-29T11:08:57.500 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQA4oQZm+K/gMBAAhA8a7qSnQSYdkUoC6wUHhg== 2024-03-29T11:08:57.500 INFO:teuthology.orchestra.run.smithi070.stdout:[mgr.x] 2024-03-29T11:08:57.500 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQAuoQZmsv4AHRAAXnB6t/t/YZed82UeRxiWPg== 2024-03-29T11:08:57.500 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.4] 2024-03-29T11:08:57.500 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQA9oQZmC68tABAAGJOOC06tLIuNULpTKz/gMg== 2024-03-29T11:08:57.500 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.5] 2024-03-29T11:08:57.500 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQA/oQZmm6ReORAAwnGxtr8KaempKsyoIZrALg== 2024-03-29T11:08:57.500 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.6] 2024-03-29T11:08:57.500 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQBCoQZmEUivNxAAxnlWAzYjfcfVScuoI2Z13g== 2024-03-29T11:08:57.501 INFO:teuthology.orchestra.run.smithi070.stdout:[osd.7] 2024-03-29T11:08:57.501 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQBFoQZm8zWdNRAA5N8j5jucRcozgN34STfZ7w== 2024-03-29T11:08:57.501 INFO:teuthology.orchestra.run.smithi070.stdout:[client.0] 2024-03-29T11:08:57.501 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQAuoQZmmLeoHxAAgsMVwY38jg2DDAcy7cbsng== 2024-03-29T11:08:57.501 INFO:teuthology.orchestra.run.smithi070.stdout:[client.1] 2024-03-29T11:08:57.501 INFO:teuthology.orchestra.run.smithi070.stdout: key = AQAuoQZm8KYhIxAARpiJLg7dC0lCRmCd6JIgDA== 2024-03-29T11:08:57.502 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:08:57.522 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:08:57.595 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:08:57.597 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:08:57.649 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:08:57.651 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:08:57.703 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:08:57.705 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:08:57.756 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:08:57.759 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:08:57.810 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:08:57.812 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:08:57.863 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:08:57.866 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:08:57.916 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:08:57.918 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:08:57.969 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:08:57.972 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:08:58.023 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:08:58.025 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:08:58.076 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:08:58.078 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:08:58.134 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:08:58.136 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:08:58.193 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T11:08:58.194 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T11:08:58.220 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:08:58.317 DEBUG:teuthology.orchestra.run.smithi018:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T11:08:58.343 DEBUG:teuthology.orchestra.run.smithi018:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T11:08:58.410 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:08:58.506 DEBUG:teuthology.orchestra.run.smithi018:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T11:08:58.533 DEBUG:teuthology.orchestra.run.smithi070:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T11:08:58.561 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:08:58.659 DEBUG:teuthology.orchestra.run.smithi070:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T11:08:58.686 DEBUG:teuthology.orchestra.run.smithi018:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:08:58.688 DEBUG:teuthology.orchestra.run.smithi070:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:08:58.741 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T11:08:58.742 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T11:08:58.742 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T11:08:58.744 INFO:tasks.ceph.mon.a:Started 2024-03-29T11:08:58.744 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T11:08:58.744 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T11:08:58.746 INFO:tasks.ceph.mon.c:Started 2024-03-29T11:08:58.746 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T11:08:58.746 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T11:08:58.784 INFO:tasks.ceph.mon.b:Started 2024-03-29T11:08:58.784 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T11:08:58.784 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T11:08:58.784 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T11:08:58.787 INFO:tasks.ceph.mgr.y:Started 2024-03-29T11:08:58.787 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T11:08:58.787 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T11:08:58.789 INFO:tasks.ceph.mgr.x:Started 2024-03-29T11:08:58.789 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T11:08:58.789 DEBUG:teuthology.orchestra.run.smithi018:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T11:08:58.891 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:58.913+0000 7fc69a1f4640 10 monclient: get_monmap_and_config 2024-03-29T11:08:58.892 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:58.913+0000 7fc69a1f4640 10 monclient: build_initial_monmap 2024-03-29T11:08:58.892 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:58.913+0000 7fc69a1f4640 10 monclient: monmap: 2024-03-29T11:08:58.892 INFO:teuthology.orchestra.run.smithi018.stderr:epoch 0 2024-03-29T11:08:58.892 INFO:teuthology.orchestra.run.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:08:58.892 INFO:teuthology.orchestra.run.smithi018.stderr:last_changed 2024-03-29T11:08:58.914467+0000 2024-03-29T11:08:58.892 INFO:teuthology.orchestra.run.smithi018.stderr:created 2024-03-29T11:08:58.914467+0000 2024-03-29T11:08:58.892 INFO:teuthology.orchestra.run.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:08:58.892 INFO:teuthology.orchestra.run.smithi018.stderr:election_strategy: 1 2024-03-29T11:08:58.892 INFO:teuthology.orchestra.run.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:08:58.892 INFO:teuthology.orchestra.run.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:08:58.893 INFO:teuthology.orchestra.run.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:08:58.893 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-29T11:08:58.894 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:58.914+0000 7fc69a1f4640 1 Processor -- start 2024-03-29T11:08:58.894 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:58.914+0000 7fc69a1f4640 1 -- start start 2024-03-29T11:08:58.894 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:58.914+0000 7fc69a1f4640 10 monclient: init 2024-03-29T11:08:58.894 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:58.914+0000 7fc69a1f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:08:58.895 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:58.914+0000 7fc69a1f4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:08:58.895 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:58.914+0000 7fc69a1f4640 10 monclient(hunting): picked mon.noname-a con 0x7fc6940b8240 addr v1:172.21.15.18:6789/0 2024-03-29T11:08:58.895 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:58.914+0000 7fc69a1f4640 10 monclient(hunting): picked mon.noname-c con 0x7fc6940b1f60 addr v1:172.21.15.70:6789/0 2024-03-29T11:08:58.895 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:58.914+0000 7fc69a1f4640 10 monclient(hunting): picked mon.noname-b con 0x7fc69409c8c0 addr v1:172.21.15.18:6790/0 2024-03-29T11:08:58.895 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:58.914+0000 7fc69a1f4640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc69409bd80 con 0x7fc6940b8240 2024-03-29T11:08:58.895 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:58.914+0000 7fc69a1f4640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc69409b960 con 0x7fc6940b1f60 2024-03-29T11:08:58.895 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:58.914+0000 7fc69a1f4640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6940b2ec0 con 0x7fc69409c8c0 2024-03-29T11:08:58.895 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:58.914+0000 7fc69a1f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:08:58.895 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:58.914+0000 7fc69a1f4640 10 monclient(hunting): authenticate will time out at 765.186882s 2024-03-29T11:08:58.895 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:58.915+0000 7fc6937fe640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fc6940b8240 0x7fc69413f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:54882/0 (socket says 172.21.15.18:54882) 2024-03-29T11:08:58.895 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:58.915+0000 7fc693fff640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fc69409c8c0 0x7fc69414d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:51442/0 (socket says 172.21.15.18:51442) 2024-03-29T11:08:58.896 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:08:58.915+0000 7fc6937fe640 1 -- 172.21.15.18:0/1306719813 learned_addr learned my addr 172.21.15.18:0/1306719813 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:09:01.894 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:01.915+0000 7fc6917fa640 10 monclient(hunting): tick 2024-03-29T11:09:01.894 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:01.915+0000 7fc6917fa640 10 monclient(hunting): _check_auth_tickets 2024-03-29T11:09:01.894 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:01.915+0000 7fc6917fa640 1 monclient(hunting): continuing hunt 2024-03-29T11:09:01.894 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:01.915+0000 7fc6917fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T11:09:01.894 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:01.915+0000 7fc6917fa640 1 -- 172.21.15.18:0/1306719813 >> v1:172.21.15.18:6790/0 conn(0x7fc69409c8c0 legacy=0x7fc69414d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:01.894 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:01.915+0000 7fc6917fa640 1 -- 172.21.15.18:0/1306719813 >> v1:172.21.15.70:6789/0 conn(0x7fc6940b1f60 legacy=0x7fc69409c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:01.894 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:01.915+0000 7fc6917fa640 1 -- 172.21.15.18:0/1306719813 >> v1:172.21.15.18:6789/0 conn(0x7fc6940b8240 legacy=0x7fc69413f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:01.894 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:01.915+0000 7fc6917fa640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:09:01.894 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:01.915+0000 7fc6917fa640 10 monclient(hunting): picked mon.noname-a con 0x7fc6800036e0 addr v1:172.21.15.18:6789/0 2024-03-29T11:09:01.895 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:01.915+0000 7fc6917fa640 10 monclient(hunting): picked mon.noname-b con 0x7fc69409c8c0 addr v1:172.21.15.18:6790/0 2024-03-29T11:09:01.895 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:01.915+0000 7fc6917fa640 10 monclient(hunting): picked mon.noname-c con 0x7fc68000aa70 addr v1:172.21.15.70:6789/0 2024-03-29T11:09:01.895 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:01.915+0000 7fc6917fa640 1 -- 172.21.15.18:0/1306719813 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc69413fb60 con 0x7fc6800036e0 2024-03-29T11:09:01.895 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:01.915+0000 7fc6917fa640 1 -- 172.21.15.18:0/1306719813 --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc68000f760 con 0x7fc68000aa70 2024-03-29T11:09:01.895 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:01.915+0000 7fc6917fa640 1 -- 172.21.15.18:0/1306719813 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6800109c0 con 0x7fc69409c8c0 2024-03-29T11:09:01.895 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:01.915+0000 7fc6917fa640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:04.057 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.078+0000 7fc6927fc640 1 -- 172.21.15.18:0/1306719813 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1502818419 0 0) 0x7fc69413fb60 con 0x7fc6800036e0 2024-03-29T11:09:04.058 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.078+0000 7fc6927fc640 10 monclient(hunting): _negotiate 2024-03-29T11:09:04.058 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.078+0000 7fc6927fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:04.058 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.078+0000 7fc6927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:04.058 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.078+0000 7fc6927fc640 10 monclient(hunting): my global_id is 4101 2024-03-29T11:09:04.058 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.078+0000 7fc6927fc640 1 -- 172.21.15.18:0/1306719813 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc67c003520 con 0x7fc6800036e0 2024-03-29T11:09:04.058 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.078+0000 7fc6927fc640 1 -- 172.21.15.18:0/1306719813 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1584807417 0 0) 0x7fc6800109c0 con 0x7fc69409c8c0 2024-03-29T11:09:04.058 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.078+0000 7fc6927fc640 10 monclient(hunting): _negotiate 2024-03-29T11:09:04.058 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.078+0000 7fc6927fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:04.058 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.078+0000 7fc6927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:04.058 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.078+0000 7fc6927fc640 10 monclient(hunting): my global_id is 4106 2024-03-29T11:09:04.058 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.078+0000 7fc6927fc640 1 -- 172.21.15.18:0/1306719813 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc69413fb60 con 0x7fc69409c8c0 2024-03-29T11:09:04.058 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.078+0000 7fc6927fc640 1 -- 172.21.15.18:0/1306719813 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1954065497 0 0) 0x7fc68000f760 con 0x7fc68000aa70 2024-03-29T11:09:04.059 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.078+0000 7fc6927fc640 10 monclient(hunting): _negotiate 2024-03-29T11:09:04.059 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.078+0000 7fc6927fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:04.059 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.078+0000 7fc6927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:04.059 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.078+0000 7fc6927fc640 10 monclient(hunting): my global_id is 4105 2024-03-29T11:09:04.059 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.078+0000 7fc6927fc640 1 -- 172.21.15.18:0/1306719813 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6800109c0 con 0x7fc68000aa70 2024-03-29T11:09:04.059 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.078+0000 7fc6927fc640 1 -- 172.21.15.18:0/1306719813 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1304870244 0 0) 0x7fc69413fb60 con 0x7fc69409c8c0 2024-03-29T11:09:04.059 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.078+0000 7fc6927fc640 1 -- 172.21.15.18:0/1306719813 >> v1:172.21.15.70:6789/0 conn(0x7fc68000aa70 legacy=0x7fc68000cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:04.059 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.078+0000 7fc6927fc640 1 -- 172.21.15.18:0/1306719813 >> v1:172.21.15.18:6789/0 conn(0x7fc6800036e0 legacy=0x7fc680005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:04.059 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.078+0000 7fc6927fc640 10 monclient: _finish_hunting 0 2024-03-29T11:09:04.059 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.078+0000 7fc6927fc640 1 monclient: found mon.noname-b 2024-03-29T11:09:04.059 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.078+0000 7fc6927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:04.059 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.078+0000 7fc6927fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:09:04.059 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.078+0000 7fc6927fc640 1 -- 172.21.15.18:0/1306719813 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc680011c20 con 0x7fc69409c8c0 2024-03-29T11:09:04.060 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.078+0000 7fc6927fc640 10 monclient: _finish_auth 0 2024-03-29T11:09:04.060 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.078+0000 7fc6927fc640 10 monclient: _check_auth_tickets 2024-03-29T11:09:04.060 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.078+0000 7fc6927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:04.060 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.078+0000 7fc6927fc640 1 -- 172.21.15.18:0/1306719813 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fc688001690 con 0x7fc69409c8c0 2024-03-29T11:09:04.060 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.078+0000 7fc6927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:04.060 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.078+0000 7fc6927fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:09:04.060 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.078+0000 7fc6927fc640 10 monclient: dump: 2024-03-29T11:09:04.060 INFO:teuthology.orchestra.run.smithi018.stderr:epoch 1 2024-03-29T11:09:04.060 INFO:teuthology.orchestra.run.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:04.060 INFO:teuthology.orchestra.run.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:04.060 INFO:teuthology.orchestra.run.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:04.060 INFO:teuthology.orchestra.run.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:04.060 INFO:teuthology.orchestra.run.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:04.060 INFO:teuthology.orchestra.run.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:04.061 INFO:teuthology.orchestra.run.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:04.061 INFO:teuthology.orchestra.run.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:04.061 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-29T11:09:04.061 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.078+0000 7fc69a1f4640 5 monclient: authenticate success, global_id 4106 2024-03-29T11:09:04.061 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.078+0000 7fc69a1f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:04.061 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.079+0000 7fc6927fc640 1 -- 172.21.15.18:0/1306719813 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc688001860 con 0x7fc69409c8c0 2024-03-29T11:09:04.061 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.079+0000 7fc6927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:04.061 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.079+0000 7fc6927fc640 1 -- 172.21.15.18:0/1306719813 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fc688001c30 con 0x7fc69409c8c0 2024-03-29T11:09:04.061 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.079+0000 7fc6927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:04.061 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.079+0000 7fc6927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:09:04.061 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.079+0000 7fc6927fc640 10 monclient: dump: 2024-03-29T11:09:04.061 INFO:teuthology.orchestra.run.smithi018.stderr:epoch 1 2024-03-29T11:09:04.061 INFO:teuthology.orchestra.run.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:04.061 INFO:teuthology.orchestra.run.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:04.062 INFO:teuthology.orchestra.run.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:04.062 INFO:teuthology.orchestra.run.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:04.062 INFO:teuthology.orchestra.run.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:04.062 INFO:teuthology.orchestra.run.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:04.062 INFO:teuthology.orchestra.run.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:04.062 INFO:teuthology.orchestra.run.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:04.062 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-29T11:09:04.062 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.079+0000 7fc69a1f4640 10 monclient: get_monmap_and_config success 2024-03-29T11:09:04.062 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.079+0000 7fc69a1f4640 4 set_mon_vals no callback set 2024-03-29T11:09:04.068 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.080+0000 7fc69a1f4640 10 monclient: shutdown 2024-03-29T11:09:04.068 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.080+0000 7fc69a1f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:04.068 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.080+0000 7fc69a1f4640 1 -- 172.21.15.18:0/1306719813 >> v1:172.21.15.18:6790/0 conn(0x7fc69409c8c0 legacy=0x7fc680009300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:04.068 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.080+0000 7fc69a1f4640 1 -- 172.21.15.18:0/1306719813 shutdown_connections 2024-03-29T11:09:04.068 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.080+0000 7fc69a1f4640 1 -- 172.21.15.18:0/1306719813 >> 172.21.15.18:0/1306719813 conn(0x7fc694066b00 msgr2=0x7fc694066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:04.068 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.080+0000 7fc69a1f4640 1 -- 172.21.15.18:0/1306719813 shutdown_connections 2024-03-29T11:09:04.068 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.080+0000 7fc69a1f4640 1 -- 172.21.15.18:0/1306719813 wait complete. 2024-03-29T11:09:04.069 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.081+0000 7fc69a1f4640 10 monclient: build_initial_monmap 2024-03-29T11:09:04.069 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.081+0000 7fc69a1f4640 10 monclient: monmap: 2024-03-29T11:09:04.069 INFO:teuthology.orchestra.run.smithi018.stderr:epoch 0 2024-03-29T11:09:04.069 INFO:teuthology.orchestra.run.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:09:04.069 INFO:teuthology.orchestra.run.smithi018.stderr:last_changed 0.000000 2024-03-29T11:09:04.069 INFO:teuthology.orchestra.run.smithi018.stderr:created 0.000000 2024-03-29T11:09:04.069 INFO:teuthology.orchestra.run.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:04.069 INFO:teuthology.orchestra.run.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:04.069 INFO:teuthology.orchestra.run.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:09:04.069 INFO:teuthology.orchestra.run.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:09:04.069 INFO:teuthology.orchestra.run.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:09:04.069 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-29T11:09:04.069 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.081+0000 7fc69a1f4640 1 librados: starting msgr at 2024-03-29T11:09:04.069 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.081+0000 7fc69a1f4640 1 librados: starting objecter 2024-03-29T11:09:04.069 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.082+0000 7fc69a1f4640 1 Processor -- start 2024-03-29T11:09:04.070 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.082+0000 7fc69a1f4640 1 -- start start 2024-03-29T11:09:04.070 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.082+0000 7fc69a1f4640 1 librados: setting wanted keys 2024-03-29T11:09:04.070 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.082+0000 7fc69a1f4640 1 librados: calling monclient init 2024-03-29T11:09:04.070 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.082+0000 7fc69a1f4640 10 monclient: init 2024-03-29T11:09:04.070 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.083+0000 7fc69a1f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:04.070 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.083+0000 7fc69a1f4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:09:04.070 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.083+0000 7fc69a1f4640 10 monclient(hunting): picked mon.noname-c con 0x7fc6800036e0 addr v1:172.21.15.18:6790/0 2024-03-29T11:09:04.070 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.083+0000 7fc69a1f4640 10 monclient(hunting): picked mon.noname-a con 0x7fc68000aa70 addr v1:172.21.15.18:6789/0 2024-03-29T11:09:04.070 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.083+0000 7fc69a1f4640 10 monclient(hunting): picked mon.noname-b con 0x7fc6940aebe0 addr v1:172.21.15.70:6789/0 2024-03-29T11:09:04.070 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.083+0000 7fc69a1f4640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6941d7240 con 0x7fc68000aa70 2024-03-29T11:09:04.070 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.083+0000 7fc69a1f4640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6940b1d20 con 0x7fc6940aebe0 2024-03-29T11:09:04.070 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.083+0000 7fc69a1f4640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6941d7e70 con 0x7fc6800036e0 2024-03-29T11:09:04.070 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.083+0000 7fc69a1f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:04.070 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.083+0000 7fc69a1f4640 10 monclient(hunting): authenticate will time out at 770.355530s 2024-03-29T11:09:04.070 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.083+0000 7fc692ffd640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fc68000aa70 0x7fc6941d7760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:54920/0 (socket says 172.21.15.18:54920) 2024-03-29T11:09:04.071 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.083+0000 7fc692ffd640 1 -- 172.21.15.18:0/2674593516 learned_addr learned my addr 172.21.15.18:0/2674593516 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:09:04.071 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.083+0000 7fc67bfff640 10 client.?.objecter ms_handle_connect 0x7fc68000aa70 2024-03-29T11:09:04.071 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.083+0000 7fc67bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:04.071 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.083+0000 7fc67bfff640 10 client.?.objecter ms_handle_connect 0x7fc6800036e0 2024-03-29T11:09:04.071 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.083+0000 7fc67bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:04.071 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.083+0000 7fc67bfff640 10 client.?.objecter ms_handle_connect 0x7fc6940aebe0 2024-03-29T11:09:04.071 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.083+0000 7fc67bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:04.071 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.084+0000 7fc67bfff640 1 -- 172.21.15.18:0/2674593516 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 659502217 0 0) 0x7fc6940b1d20 con 0x7fc6940aebe0 2024-03-29T11:09:04.071 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.084+0000 7fc67bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:09:04.071 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.084+0000 7fc67bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:04.071 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.084+0000 7fc67bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:04.071 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.084+0000 7fc67bfff640 10 monclient(hunting): my global_id is 4108 2024-03-29T11:09:04.071 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.084+0000 7fc67bfff640 1 -- 172.21.15.18:0/2674593516 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc68000a2d0 con 0x7fc6940aebe0 2024-03-29T11:09:04.071 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.084+0000 7fc67bfff640 1 -- 172.21.15.18:0/2674593516 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 127181027 0 0) 0x7fc6941d7e70 con 0x7fc6800036e0 2024-03-29T11:09:04.071 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.084+0000 7fc67bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:09:04.072 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.084+0000 7fc67bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:04.072 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.084+0000 7fc67bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:04.072 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.084+0000 7fc67bfff640 10 monclient(hunting): my global_id is 4112 2024-03-29T11:09:04.072 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.084+0000 7fc67bfff640 1 -- 172.21.15.18:0/2674593516 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6940b1d20 con 0x7fc6800036e0 2024-03-29T11:09:04.072 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.084+0000 7fc67bfff640 1 -- 172.21.15.18:0/2674593516 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3199244528 0 0) 0x7fc68000a2d0 con 0x7fc6940aebe0 2024-03-29T11:09:04.072 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.084+0000 7fc67bfff640 1 -- 172.21.15.18:0/2674593516 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc6941d7e70 con 0x7fc6940aebe0 2024-03-29T11:09:04.072 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.084+0000 7fc67bfff640 1 -- 172.21.15.18:0/2674593516 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fc68c0017e0 con 0x7fc6940aebe0 2024-03-29T11:09:04.072 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.084+0000 7fc67bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:04.072 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.084+0000 7fc67bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:09:04.072 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.084+0000 7fc67bfff640 10 monclient(hunting): dump: 2024-03-29T11:09:04.072 INFO:teuthology.orchestra.run.smithi018.stderr:epoch 1 2024-03-29T11:09:04.072 INFO:teuthology.orchestra.run.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:04.072 INFO:teuthology.orchestra.run.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:04.072 INFO:teuthology.orchestra.run.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:04.072 INFO:teuthology.orchestra.run.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:04.072 INFO:teuthology.orchestra.run.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:04.073 INFO:teuthology.orchestra.run.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:04.073 INFO:teuthology.orchestra.run.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:04.073 INFO:teuthology.orchestra.run.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:04.073 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-29T11:09:04.073 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.084+0000 7fc67bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:09:04.073 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.084+0000 7fc67bfff640 1 -- 172.21.15.18:0/2674593516 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 501230744 0 0) 0x7fc6940b1d20 con 0x7fc6800036e0 2024-03-29T11:09:04.073 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.084+0000 7fc67bfff640 1 -- 172.21.15.18:0/2674593516 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc68000a2d0 con 0x7fc6800036e0 2024-03-29T11:09:04.073 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.084+0000 7fc67bfff640 1 -- 172.21.15.18:0/2674593516 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fc684001c70 con 0x7fc6800036e0 2024-03-29T11:09:04.073 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.084+0000 7fc67bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:04.073 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.084+0000 7fc67bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:09:04.073 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.084+0000 7fc67bfff640 10 monclient(hunting): dump: 2024-03-29T11:09:04.073 INFO:teuthology.orchestra.run.smithi018.stderr:epoch 1 2024-03-29T11:09:04.073 INFO:teuthology.orchestra.run.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:04.073 INFO:teuthology.orchestra.run.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:04.074 INFO:teuthology.orchestra.run.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:04.074 INFO:teuthology.orchestra.run.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:04.074 INFO:teuthology.orchestra.run.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:04.074 INFO:teuthology.orchestra.run.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:04.074 INFO:teuthology.orchestra.run.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:04.074 INFO:teuthology.orchestra.run.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:04.074 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-29T11:09:04.074 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.084+0000 7fc67bfff640 1 -- 172.21.15.18:0/2674593516 <== mon.1 v1:172.21.15.70:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2160337157 0 0) 0x7fc6941d7e70 con 0x7fc6940aebe0 2024-03-29T11:09:04.074 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.084+0000 7fc67bfff640 1 -- 172.21.15.18:0/2674593516 >> v1:172.21.15.18:6790/0 conn(0x7fc6800036e0 legacy=0x7fc6940b1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:04.074 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.084+0000 7fc67bfff640 1 -- 172.21.15.18:0/2674593516 >> v1:172.21.15.18:6789/0 conn(0x7fc68000aa70 legacy=0x7fc6941d7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:04.074 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.084+0000 7fc67bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:09:04.074 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.084+0000 7fc67bfff640 1 monclient: found mon.b 2024-03-29T11:09:04.074 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.084+0000 7fc67bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:04.074 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.084+0000 7fc67bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:04.075 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.084+0000 7fc67bfff640 1 -- 172.21.15.18:0/2674593516 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6941d8000 con 0x7fc6940aebe0 2024-03-29T11:09:04.075 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.084+0000 7fc67bfff640 10 monclient: _finish_auth 0 2024-03-29T11:09:04.075 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.084+0000 7fc67bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:09:04.075 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.084+0000 7fc67bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:04.075 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.084+0000 7fc69a1f4640 5 monclient: authenticate success, global_id 4108 2024-03-29T11:09:04.075 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.085+0000 7fc69a1f4640 10 monclient: _renew_subs 2024-03-29T11:09:04.075 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.085+0000 7fc69a1f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:04.075 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.085+0000 7fc69a1f4640 1 -- 172.21.15.18:0/2674593516 --> v1:172.21.15.70:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc69411fb30 con 0x7fc6940aebe0 2024-03-29T11:09:04.075 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.085+0000 7fc69a1f4640 10 client.4108.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:09:04.075 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.085+0000 7fc67bfff640 1 -- 172.21.15.18:0/2674593516 <== mon.1 v1:172.21.15.70:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc68c001cd0 con 0x7fc6940aebe0 2024-03-29T11:09:04.075 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.085+0000 7fc67bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:04.075 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.085+0000 7fc691ffb640 4 set_mon_vals no callback set 2024-03-29T11:09:04.075 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.085+0000 7fc67bfff640 1 -- 172.21.15.18:0/2674593516 <== mon.1 v1:172.21.15.70:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fc68c004980 con 0x7fc6940aebe0 2024-03-29T11:09:04.076 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.085+0000 7fc69a1f4640 10 monclient: _renew_subs 2024-03-29T11:09:04.076 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.085+0000 7fc69a1f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:04.076 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.085+0000 7fc69a1f4640 1 -- 172.21.15.18:0/2674593516 --> v1:172.21.15.70:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc69413b600 con 0x7fc6940aebe0 2024-03-29T11:09:04.076 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.085+0000 7fc69a1f4640 1 librados: init done 2024-03-29T11:09:04.076 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.085+0000 7fc67bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:04.076 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.085+0000 7fc67bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:09:04.076 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.085+0000 7fc67bfff640 10 monclient: dump: 2024-03-29T11:09:04.076 INFO:teuthology.orchestra.run.smithi018.stderr:epoch 1 2024-03-29T11:09:04.076 INFO:teuthology.orchestra.run.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:04.076 INFO:teuthology.orchestra.run.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:04.076 INFO:teuthology.orchestra.run.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:04.076 INFO:teuthology.orchestra.run.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:04.076 INFO:teuthology.orchestra.run.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:04.076 INFO:teuthology.orchestra.run.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:04.076 INFO:teuthology.orchestra.run.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:04.076 INFO:teuthology.orchestra.run.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:04.077 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-29T11:09:04.077 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.085+0000 7fc67bfff640 1 -- 172.21.15.18:0/2674593516 <== mon.1 v1:172.21.15.70:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc68c004e70 con 0x7fc6940aebe0 2024-03-29T11:09:04.077 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.085+0000 7fc67bfff640 1 -- 172.21.15.18:0/2674593516 <== mon.1 v1:172.21.15.70:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 1999126531 0 0) 0x7fc68c006380 con 0x7fc6940aebe0 2024-03-29T11:09:04.077 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.085+0000 7fc67bfff640 10 client.4108.objecter ms_dispatch 0x7fc694000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T11:09:04.077 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.085+0000 7fc67bfff640 3 client.4108.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T11:09:04.077 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.085+0000 7fc67bfff640 3 client.4108.objecter handle_osd_map decoding full epoch 1 2024-03-29T11:09:04.077 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.085+0000 7fc67bfff640 20 client.4108.objecter dump_active .. 0 homeless 2024-03-29T11:09:04.077 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.086+0000 7fc69a1f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:04.077 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.086+0000 7fc69a1f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:04.077 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.086+0000 7fc69a1f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:04.077 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.086+0000 7fc69a1f4640 1 -- 172.21.15.18:0/2674593516 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc694121380 con 0x7fc6940aebe0 2024-03-29T11:09:04.077 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.088+0000 7fc67bfff640 1 -- 172.21.15.18:0/2674593516 <== mon.1 v1:172.21.15.70:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fc68c005370 con 0x7fc6940aebe0 2024-03-29T11:09:04.077 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.088+0000 7fc67bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:04.077 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.088+0000 7fc67bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:09:04.145 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.167+0000 7fc69a1f4640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:09:04.145 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.167+0000 7fc69a1f4640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:09:04.145 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.167+0000 7fc69a1f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:04.145 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.167+0000 7fc69a1f4640 1 -- 172.21.15.18:0/2674593516 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fc69404f730 con 0x7fc6940aebe0 2024-03-29T11:09:04.149 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.170+0000 7fc67bfff640 1 -- 172.21.15.18:0/2674593516 <== mon.1 v1:172.21.15.70:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7fc68c005670 con 0x7fc6940aebe0 2024-03-29T11:09:04.149 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.170+0000 7fc67bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:09:04.149 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.170+0000 7fc67bfff640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T11:09:04.149 INFO:teuthology.orchestra.run.smithi018.stderr:adjusted tunables profile to default 2024-03-29T11:09:04.150 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.171+0000 7fc69a1f4640 10 librados: watch_flush enter 2024-03-29T11:09:04.150 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.171+0000 7fc69a1f4640 10 librados: watch_flush exit 2024-03-29T11:09:04.150 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.171+0000 7fc69a1f4640 20 client.4108.objecter shutdown clearing up homeless session... 2024-03-29T11:09:04.150 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.171+0000 7fc69a1f4640 10 client.4108.objecter successfully canceled tick 2024-03-29T11:09:04.150 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.171+0000 7fc69a1f4640 10 monclient: shutdown 2024-03-29T11:09:04.150 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.171+0000 7fc69a1f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:04.150 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.171+0000 7fc69a1f4640 1 -- 172.21.15.18:0/2674593516 >> v1:172.21.15.70:6789/0 conn(0x7fc6940aebe0 legacy=0x7fc69413aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:04.151 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.171+0000 7fc69a1f4640 1 -- 172.21.15.18:0/2674593516 shutdown_connections 2024-03-29T11:09:04.151 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.171+0000 7fc69a1f4640 1 -- 172.21.15.18:0/2674593516 >> 172.21.15.18:0/2674593516 conn(0x7fc694066b00 msgr2=0x7fc6940b6de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:04.151 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.171+0000 7fc69a1f4640 1 -- 172.21.15.18:0/2674593516 shutdown_connections 2024-03-29T11:09:04.151 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.171+0000 7fc69a1f4640 1 -- 172.21.15.18:0/2674593516 wait complete. 2024-03-29T11:09:04.151 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:04.171+0000 7fc69a1f4640 1 librados: shutdown 2024-03-29T11:09:04.161 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T11:09:04.161 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T11:09:04.161 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-29T11:09:04.161 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T11:09:04.188 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-29T11:09:04.189 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T11:09:04.256 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-29T11:09:04.256 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T11:09:04.323 DEBUG:teuthology.orchestra.run.smithi018:> set -ex 2024-03-29T11:09:04.323 DEBUG:teuthology.orchestra.run.smithi018:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T11:09:04.390 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-03-29T11:09:04.390 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T11:09:04.417 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-03-29T11:09:04.417 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T11:09:04.484 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-03-29T11:09:04.484 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T11:09:04.551 DEBUG:teuthology.orchestra.run.smithi070:> set -ex 2024-03-29T11:09:04.551 DEBUG:teuthology.orchestra.run.smithi070:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T11:09:04.578 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-29T11:09:04.599+0000 7fad3de2e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:09:04.600 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-03-29T11:09:04.622+0000 7f6963a2c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:09:04.618 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --cluster ceph osd new 02a13c46-dd11-4de5-8fbd-bb4aadf09403 0 2024-03-29T11:09:04.634 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-29T11:09:04.656+0000 7fad3de2e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:09:04.656 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-03-29T11:09:04.677+0000 7f6963a2c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:09:04.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.776+0000 7f62d0c4e640 10 monclient: get_monmap_and_config 2024-03-29T11:09:04.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.776+0000 7f62d0c4e640 10 monclient: build_initial_monmap 2024-03-29T11:09:04.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.776+0000 7f62d0c4e640 10 monclient: monmap: 2024-03-29T11:09:04.755 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-03-29T11:09:04.755 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:04.755 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:09:04.777660+0000 2024-03-29T11:09:04.756 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:09:04.777660+0000 2024-03-29T11:09:04.756 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:04.756 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:04.756 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:09:04.756 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:09:04.756 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:09:04.756 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:04.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.777+0000 7f62d0c4e640 1 Processor -- start 2024-03-29T11:09:04.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.777+0000 7f62d0c4e640 1 -- start start 2024-03-29T11:09:04.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.777+0000 7f62d0c4e640 10 monclient: init 2024-03-29T11:09:04.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.777+0000 7f62d0c4e640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:04.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.777+0000 7f62d0c4e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:09:04.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.777+0000 7f62d0c4e640 10 monclient(hunting): picked mon.noname-a con 0x7f62cc09ac40 addr v1:172.21.15.18:6789/0 2024-03-29T11:09:04.765 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.777+0000 7f62d0c4e640 10 monclient(hunting): picked mon.noname-c con 0x7f62cc09c870 addr v1:172.21.15.70:6789/0 2024-03-29T11:09:04.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.777+0000 7f62d0c4e640 10 monclient(hunting): picked mon.noname-b con 0x7f62cc0b4360 addr v1:172.21.15.18:6790/0 2024-03-29T11:09:04.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.777+0000 7f62d0c4e640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62cc09dec0 con 0x7f62cc09ac40 2024-03-29T11:09:04.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.777+0000 7f62d0c4e640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62cc09da30 con 0x7f62cc09c870 2024-03-29T11:09:04.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.777+0000 7f62d0c4e640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62cc0b39c0 con 0x7f62cc0b4360 2024-03-29T11:09:04.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.777+0000 7f62d0c4e640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:04.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.777+0000 7f62d0c4e640 10 monclient(hunting): authenticate will time out at 761.006189s 2024-03-29T11:09:04.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c3fff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f62cc09c870 0x7f62cc13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:48194/0 (socket says 172.21.15.70:48194) 2024-03-29T11:09:04.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c3fff640 1 -- 172.21.15.70:0/2246198045 learned_addr learned my addr 172.21.15.70:0/2246198045 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-03-29T11:09:04.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c9d9b640 1 -- 172.21.15.70:0/2246198045 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 574155861 0 0) 0x7f62cc09da30 con 0x7f62cc09c870 2024-03-29T11:09:04.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c9d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:09:04.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:04.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:04.766 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c9d9b640 10 monclient(hunting): my global_id is 4111 2024-03-29T11:09:04.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c9d9b640 1 -- 172.21.15.70:0/2246198045 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62ac003520 con 0x7f62cc09c870 2024-03-29T11:09:04.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c9d9b640 1 -- 172.21.15.70:0/2246198045 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3444154979 0 0) 0x7f62cc0b39c0 con 0x7f62cc0b4360 2024-03-29T11:09:04.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c9d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:09:04.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:04.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:04.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c9d9b640 10 monclient(hunting): my global_id is 4115 2024-03-29T11:09:04.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c9d9b640 1 -- 172.21.15.70:0/2246198045 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62cc09da30 con 0x7f62cc0b4360 2024-03-29T11:09:04.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c9d9b640 1 -- 172.21.15.70:0/2246198045 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3964292705 0 0) 0x7f62bc001110 con 0x7f62cc09ac40 2024-03-29T11:09:04.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c9d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:09:04.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:04.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:04.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c9d9b640 10 monclient(hunting): my global_id is 4113 2024-03-29T11:09:04.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c9d9b640 1 -- 172.21.15.70:0/2246198045 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62cc0b39c0 con 0x7f62cc09ac40 2024-03-29T11:09:04.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c9d9b640 1 -- 172.21.15.70:0/2246198045 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 650897497 0 0) 0x7f62ac003520 con 0x7f62cc09c870 2024-03-29T11:09:04.767 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c9d9b640 1 -- 172.21.15.70:0/2246198045 >> v1:172.21.15.18:6790/0 conn(0x7f62cc0b4360 legacy=0x7f62cc14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:04.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c9d9b640 1 -- 172.21.15.70:0/2246198045 >> v1:172.21.15.18:6789/0 conn(0x7f62cc09ac40 legacy=0x7f62cc0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:04.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c9d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:09:04.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c9d9b640 1 monclient: found mon.noname-c 2024-03-29T11:09:04.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:04.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c9d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.70:6789/0 2024-03-29T11:09:04.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c9d9b640 1 -- 172.21.15.70:0/2246198045 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62cc0b5160 con 0x7f62cc09c870 2024-03-29T11:09:04.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c9d9b640 10 monclient: _finish_auth 0 2024-03-29T11:09:04.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c9d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:09:04.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:04.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62d0c4e640 5 monclient: authenticate success, global_id 4111 2024-03-29T11:09:04.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62d0c4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:04.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c9d9b640 1 -- 172.21.15.70:0/2246198045 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f62b4001690 con 0x7f62cc09c870 2024-03-29T11:09:04.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:04.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c9d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:09:04.768 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c9d9b640 10 monclient: dump: 2024-03-29T11:09:04.769 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:04.769 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:04.769 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:04.769 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:04.769 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:04.769 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:04.769 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:04.769 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:04.769 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:04.769 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:04.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62d0c4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:04.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c9d9b640 1 -- 172.21.15.70:0/2246198045 <== mon.1 v1:172.21.15.70:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62b4001870 con 0x7f62cc09c870 2024-03-29T11:09:04.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:04.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c9d9b640 1 -- 172.21.15.70:0/2246198045 <== mon.1 v1:172.21.15.70:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f62b4001c80 con 0x7f62cc09c870 2024-03-29T11:09:04.769 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62d0c4e640 10 monclient: get_monmap_and_config success 2024-03-29T11:09:04.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62d0c4e640 4 set_mon_vals no callback set 2024-03-29T11:09:04.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:04.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c9d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:09:04.770 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.778+0000 7f62c9d9b640 10 monclient: dump: 2024-03-29T11:09:04.770 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:04.770 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:04.770 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:04.770 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:04.770 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:04.770 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:04.770 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:04.770 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:04.770 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:04.770 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:04.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.779+0000 7f62d0c4e640 10 monclient: shutdown 2024-03-29T11:09:04.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.779+0000 7f62d0c4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:04.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.779+0000 7f62d0c4e640 1 -- 172.21.15.70:0/2246198045 >> v1:172.21.15.70:6789/0 conn(0x7f62cc09c870 legacy=0x7f62cc13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:04.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.779+0000 7f62d0c4e640 1 -- 172.21.15.70:0/2246198045 shutdown_connections 2024-03-29T11:09:04.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.779+0000 7f62d0c4e640 1 -- 172.21.15.70:0/2246198045 >> 172.21.15.70:0/2246198045 conn(0x7f62cc066b00 msgr2=0x7f62cc066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:04.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.779+0000 7f62d0c4e640 1 -- 172.21.15.70:0/2246198045 shutdown_connections 2024-03-29T11:09:04.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.779+0000 7f62d0c4e640 1 -- 172.21.15.70:0/2246198045 wait complete. 2024-03-29T11:09:04.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.780+0000 7f62d0c4e640 10 monclient: build_initial_monmap 2024-03-29T11:09:04.771 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.780+0000 7f62d0c4e640 10 monclient: monmap: 2024-03-29T11:09:04.771 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-03-29T11:09:04.771 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:09:04.771 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-03-29T11:09:04.771 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-03-29T11:09:04.771 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:04.771 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:04.772 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:09:04.772 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:09:04.772 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:09:04.772 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:04.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.780+0000 7f62d0c4e640 1 librados: starting msgr at 2024-03-29T11:09:04.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.780+0000 7f62d0c4e640 1 librados: starting objecter 2024-03-29T11:09:04.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.780+0000 7f62d0c4e640 1 Processor -- start 2024-03-29T11:09:04.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.780+0000 7f62d0c4e640 1 -- start start 2024-03-29T11:09:04.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.780+0000 7f62d0c4e640 1 librados: setting wanted keys 2024-03-29T11:09:04.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.780+0000 7f62d0c4e640 1 librados: calling monclient init 2024-03-29T11:09:04.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.780+0000 7f62d0c4e640 10 monclient: init 2024-03-29T11:09:04.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62d0c4e640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:04.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62d0c4e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:09:04.772 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62d0c4e640 10 monclient(hunting): picked mon.noname-c con 0x7f62cc09ac40 addr v1:172.21.15.18:6790/0 2024-03-29T11:09:04.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62d0c4e640 10 monclient(hunting): picked mon.noname-a con 0x7f62cc0b07a0 addr v1:172.21.15.18:6789/0 2024-03-29T11:09:04.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62d0c4e640 10 monclient(hunting): picked mon.noname-b con 0x7f62cc1d7cd0 addr v1:172.21.15.70:6789/0 2024-03-29T11:09:04.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62d0c4e640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62cc0acd70 con 0x7f62cc0b07a0 2024-03-29T11:09:04.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62d0c4e640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62cc1d8100 con 0x7f62cc1d7cd0 2024-03-29T11:09:04.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62d0c4e640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62cc11f8d0 con 0x7f62cc09ac40 2024-03-29T11:09:04.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62d0c4e640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:04.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62d0c4e640 10 monclient(hunting): authenticate will time out at 761.009325s 2024-03-29T11:09:04.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62cad9d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f62cc1d7cd0 0x7f62cc1e1d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:48196/0 (socket says 172.21.15.70:48196) 2024-03-29T11:09:04.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62cad9d640 1 -- 172.21.15.70:0/2254279926 learned_addr learned my addr 172.21.15.70:0/2254279926 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-03-29T11:09:04.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 10 client.?.objecter ms_handle_connect 0x7f62cc1d7cd0 2024-03-29T11:09:04.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:04.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 10 client.?.objecter ms_handle_connect 0x7f62cc09ac40 2024-03-29T11:09:04.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:04.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 10 client.?.objecter ms_handle_connect 0x7f62cc0b07a0 2024-03-29T11:09:04.773 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:04.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 1 -- 172.21.15.70:0/2254279926 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 536196820 0 0) 0x7f62cc1d8100 con 0x7f62cc1d7cd0 2024-03-29T11:09:04.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:09:04.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:04.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:04.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 10 monclient(hunting): my global_id is 4114 2024-03-29T11:09:04.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 1 -- 172.21.15.70:0/2254279926 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62b0004eb0 con 0x7f62cc1d7cd0 2024-03-29T11:09:04.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 1 -- 172.21.15.70:0/2254279926 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1250725077 0 0) 0x7f62cc11f8d0 con 0x7f62cc09ac40 2024-03-29T11:09:04.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:09:04.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:04.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:04.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 10 monclient(hunting): my global_id is 4118 2024-03-29T11:09:04.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 1 -- 172.21.15.70:0/2254279926 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62cc1d8100 con 0x7f62cc09ac40 2024-03-29T11:09:04.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 1 -- 172.21.15.70:0/2254279926 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3597056527 0 0) 0x7f62cc0acd70 con 0x7f62cc0b07a0 2024-03-29T11:09:04.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:09:04.774 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:04.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:04.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 10 monclient(hunting): my global_id is 4116 2024-03-29T11:09:04.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 1 -- 172.21.15.70:0/2254279926 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62cc11f8d0 con 0x7f62cc0b07a0 2024-03-29T11:09:04.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 1 -- 172.21.15.70:0/2254279926 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 603386895 0 0) 0x7f62b0004eb0 con 0x7f62cc1d7cd0 2024-03-29T11:09:04.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 1 -- 172.21.15.70:0/2254279926 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62cc0acd70 con 0x7f62cc1d7cd0 2024-03-29T11:09:04.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 1 -- 172.21.15.70:0/2254279926 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f62c4001c90 con 0x7f62cc1d7cd0 2024-03-29T11:09:04.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:04.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:09:04.775 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 10 monclient(hunting): dump: 2024-03-29T11:09:04.775 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:04.775 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:04.775 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:04.775 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:04.775 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:04.776 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:04.776 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:04.776 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:04.776 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:04.776 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:04.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:09:04.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 1 -- 172.21.15.70:0/2254279926 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1639920445 0 0) 0x7f62cc1d8100 con 0x7f62cc09ac40 2024-03-29T11:09:04.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 1 -- 172.21.15.70:0/2254279926 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62b0004eb0 con 0x7f62cc09ac40 2024-03-29T11:09:04.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 1 -- 172.21.15.70:0/2254279926 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f62bc001760 con 0x7f62cc09ac40 2024-03-29T11:09:04.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:04.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:09:04.776 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 10 monclient(hunting): dump: 2024-03-29T11:09:04.776 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:04.776 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:04.776 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:04.777 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:04.777 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:04.777 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:04.777 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:04.777 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:04.777 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:04.777 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:04.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 1 -- 172.21.15.70:0/2254279926 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1915105751 0 0) 0x7f62cc11f8d0 con 0x7f62cc0b07a0 2024-03-29T11:09:04.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 1 -- 172.21.15.70:0/2254279926 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62cc1d8100 con 0x7f62cc0b07a0 2024-03-29T11:09:04.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 1 -- 172.21.15.70:0/2254279926 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f62b4001b80 con 0x7f62cc0b07a0 2024-03-29T11:09:04.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:04.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:09:04.777 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 10 monclient(hunting): dump: 2024-03-29T11:09:04.777 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:04.778 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:04.778 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:04.778 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:04.778 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:04.778 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:04.778 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:04.778 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:04.778 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:04.778 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:04.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 1 -- 172.21.15.70:0/2254279926 <== mon.1 v1:172.21.15.70:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 318584459 0 0) 0x7f62cc0acd70 con 0x7f62cc1d7cd0 2024-03-29T11:09:04.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 1 -- 172.21.15.70:0/2254279926 >> v1:172.21.15.18:6790/0 conn(0x7f62cc09ac40 legacy=0x7f62cc0afa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:04.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 1 -- 172.21.15.70:0/2254279926 >> v1:172.21.15.18:6789/0 conn(0x7f62cc0b07a0 legacy=0x7f62cc1d75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:04.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:09:04.778 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 1 monclient: found mon.b 2024-03-29T11:09:04.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:04.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:04.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 1 -- 172.21.15.70:0/2254279926 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62cc1e64c0 con 0x7f62cc1d7cd0 2024-03-29T11:09:04.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:09:04.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:09:04.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:04.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62d0c4e640 5 monclient: authenticate success, global_id 4114 2024-03-29T11:09:04.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62d0c4e640 10 monclient: _renew_subs 2024-03-29T11:09:04.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62d0c4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:04.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62d0c4e640 1 -- 172.21.15.70:0/2254279926 --> v1:172.21.15.70:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62cc1e3490 con 0x7f62cc1d7cd0 2024-03-29T11:09:04.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62d0c4e640 10 client.4114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:09:04.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62d0c4e640 10 monclient: _renew_subs 2024-03-29T11:09:04.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62d0c4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:04.779 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62d0c4e640 1 -- 172.21.15.70:0/2254279926 --> v1:172.21.15.70:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f62cc1e3a10 con 0x7f62cc1d7cd0 2024-03-29T11:09:04.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.781+0000 7f62d0c4e640 1 librados: init done 2024-03-29T11:09:04.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.782+0000 7f62c2ffd640 1 -- 172.21.15.70:0/2254279926 <== mon.1 v1:172.21.15.70:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f62c4003970 con 0x7f62cc1d7cd0 2024-03-29T11:09:04.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.782+0000 7f62c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:04.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.782+0000 7f62c2ffd640 1 -- 172.21.15.70:0/2254279926 <== mon.1 v1:172.21.15.70:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f62c4003cc0 con 0x7f62cc1d7cd0 2024-03-29T11:09:04.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.782+0000 7f62c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:04.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.782+0000 7f62c2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:09:04.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.782+0000 7f62c959a640 4 set_mon_vals no callback set 2024-03-29T11:09:04.780 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.782+0000 7f62c2ffd640 10 monclient: dump: 2024-03-29T11:09:04.780 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:04.780 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:04.780 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:04.780 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:04.780 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:04.780 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:04.780 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:04.781 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:04.781 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:04.781 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:04.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.782+0000 7f62c2ffd640 1 -- 172.21.15.70:0/2254279926 <== mon.1 v1:172.21.15.70:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f62c4004230 con 0x7f62cc1d7cd0 2024-03-29T11:09:04.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.782+0000 7f62c2ffd640 1 -- 172.21.15.70:0/2254279926 <== mon.1 v1:172.21.15.70:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 378275733 0 0) 0x7f62c4006980 con 0x7f62cc1d7cd0 2024-03-29T11:09:04.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.782+0000 7f62c2ffd640 10 client.4114.objecter ms_dispatch 0x7f62cc000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T11:09:04.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.782+0000 7f62c2ffd640 3 client.4114.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T11:09:04.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.782+0000 7f62c2ffd640 3 client.4114.objecter handle_osd_map decoding full epoch 2 2024-03-29T11:09:04.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.782+0000 7f62c2ffd640 20 client.4114.objecter dump_active .. 0 homeless 2024-03-29T11:09:04.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.782+0000 7f62d0c4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:04.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.782+0000 7f62d0c4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:04.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.782+0000 7f62d0c4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:04.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.782+0000 7f62d0c4e640 1 -- 172.21.15.70:0/2254279926 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62a4005080 con 0x7f62cc1d7cd0 2024-03-29T11:09:04.781 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.784+0000 7f62c2ffd640 1 -- 172.21.15.70:0/2254279926 <== mon.1 v1:172.21.15.70:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f62c40046c0 con 0x7f62cc1d7cd0 2024-03-29T11:09:04.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.784+0000 7f62c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:04.782 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.784+0000 7f62c2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:09:04.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.855+0000 7f62d0c4e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "02a13c46-dd11-4de5-8fbd-bb4aadf09403", "id": 0}] 2024-03-29T11:09:04.833 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.855+0000 7f62d0c4e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "02a13c46-dd11-4de5-8fbd-bb4aadf09403", "id": 0}] 2024-03-29T11:09:04.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.855+0000 7f62d0c4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:04.834 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.855+0000 7f62d0c4e640 1 -- 172.21.15.70:0/2254279926 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "osd new", "uuid": "02a13c46-dd11-4de5-8fbd-bb4aadf09403", "id": 0} v 0) v1 -- 0x7f62a40058b0 con 0x7f62cc1d7cd0 2024-03-29T11:09:04.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.857+0000 7f62c2ffd640 1 -- 172.21.15.70:0/2254279926 <== mon.1 v1:172.21.15.70:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "02a13c46-dd11-4de5-8fbd-bb4aadf09403", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 508722883 0 732707515) 0x7f62c4005b40 con 0x7f62cc1d7cd0 2024-03-29T11:09:04.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.857+0000 7f62c2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "02a13c46-dd11-4de5-8fbd-bb4aadf09403", "id": 0}] 2024-03-29T11:09:04.836 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.857+0000 7f62c2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:09:04.836 INFO:teuthology.orchestra.run.smithi070.stdout:0 2024-03-29T11:09:04.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.858+0000 7f62d0c4e640 10 librados: watch_flush enter 2024-03-29T11:09:04.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.858+0000 7f62d0c4e640 10 librados: watch_flush exit 2024-03-29T11:09:04.837 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.858+0000 7f62d0c4e640 20 client.4114.objecter shutdown clearing up homeless session... 2024-03-29T11:09:04.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.858+0000 7f62d0c4e640 10 client.4114.objecter successfully canceled tick 2024-03-29T11:09:04.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.858+0000 7f62d0c4e640 10 monclient: shutdown 2024-03-29T11:09:04.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.858+0000 7f62d0c4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:04.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.858+0000 7f62d0c4e640 1 -- 172.21.15.70:0/2254279926 >> v1:172.21.15.70:6789/0 conn(0x7f62cc1d7cd0 legacy=0x7f62cc1e1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:04.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.858+0000 7f62d0c4e640 1 -- 172.21.15.70:0/2254279926 shutdown_connections 2024-03-29T11:09:04.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.858+0000 7f62d0c4e640 1 -- 172.21.15.70:0/2254279926 >> 172.21.15.70:0/2254279926 conn(0x7f62cc066b00 msgr2=0x7f62cc0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:04.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.858+0000 7f62d0c4e640 1 -- 172.21.15.70:0/2254279926 shutdown_connections 2024-03-29T11:09:04.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.858+0000 7f62d0c4e640 1 -- 172.21.15.70:0/2254279926 wait complete. 2024-03-29T11:09:04.838 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.858+0000 7f62d0c4e640 1 librados: shutdown 2024-03-29T11:09:04.847 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --cluster ceph osd new 4a8fc337-370b-4c2a-83d7-d5b6e9fb2436 1 2024-03-29T11:09:04.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.966+0000 7f6f80e4e640 10 monclient: get_monmap_and_config 2024-03-29T11:09:04.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.966+0000 7f6f80e4e640 10 monclient: build_initial_monmap 2024-03-29T11:09:04.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.966+0000 7f6f80e4e640 10 monclient: monmap: 2024-03-29T11:09:04.945 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-03-29T11:09:04.945 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:04.945 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:09:04.967790+0000 2024-03-29T11:09:04.946 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:09:04.967790+0000 2024-03-29T11:09:04.946 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:04.946 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:04.946 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:09:04.946 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:09:04.946 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:09:04.946 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:04.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.967+0000 7f6f80e4e640 1 Processor -- start 2024-03-29T11:09:04.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.967+0000 7f6f80e4e640 1 -- start start 2024-03-29T11:09:04.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.967+0000 7f6f80e4e640 10 monclient: init 2024-03-29T11:09:04.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f80e4e640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:04.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f80e4e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:09:04.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f80e4e640 10 monclient(hunting): picked mon.noname-b con 0x7f6f7c09ac40 addr v1:172.21.15.18:6790/0 2024-03-29T11:09:04.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f80e4e640 10 monclient(hunting): picked mon.noname-a con 0x7f6f7c09c870 addr v1:172.21.15.18:6789/0 2024-03-29T11:09:04.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f80e4e640 10 monclient(hunting): picked mon.noname-c con 0x7f6f7c0b4360 addr v1:172.21.15.70:6789/0 2024-03-29T11:09:04.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f80e4e640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f7c09dec0 con 0x7f6f7c09c870 2024-03-29T11:09:04.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f80e4e640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f7c09da30 con 0x7f6f7c0b4360 2024-03-29T11:09:04.955 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f80e4e640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f7c0b39c0 con 0x7f6f7c09ac40 2024-03-29T11:09:04.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f80e4e640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:04.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f80e4e640 10 monclient(hunting): authenticate will time out at 761.196342s 2024-03-29T11:09:04.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f7ad9d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f6f7c0b4360 0x7f6f7c14d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:48206/0 (socket says 172.21.15.70:48206) 2024-03-29T11:09:04.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f7ad9d640 1 -- 172.21.15.70:0/4248407043 learned_addr learned my addr 172.21.15.70:0/4248407043 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-03-29T11:09:04.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f79d9b640 1 -- 172.21.15.70:0/4248407043 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2632235626 0 0) 0x7f6f7c09da30 con 0x7f6f7c0b4360 2024-03-29T11:09:04.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f79d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:09:04.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f79d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:04.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f79d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:04.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f79d9b640 10 monclient(hunting): my global_id is 4117 2024-03-29T11:09:04.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f79d9b640 1 -- 172.21.15.70:0/4248407043 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f68003520 con 0x7f6f7c0b4360 2024-03-29T11:09:04.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f79d9b640 1 -- 172.21.15.70:0/4248407043 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3037323595 0 0) 0x7f6f64001110 con 0x7f6f7c09c870 2024-03-29T11:09:04.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f79d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:09:04.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f79d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:04.956 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f79d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:04.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f79d9b640 10 monclient(hunting): my global_id is 4119 2024-03-29T11:09:04.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f79d9b640 1 -- 172.21.15.70:0/4248407043 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f7c09da30 con 0x7f6f7c09c870 2024-03-29T11:09:04.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f79d9b640 1 -- 172.21.15.70:0/4248407043 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 728553057 0 0) 0x7f6f7c0b39c0 con 0x7f6f7c09ac40 2024-03-29T11:09:04.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f79d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:09:04.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f79d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:04.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f79d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:04.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f79d9b640 10 monclient(hunting): my global_id is 4121 2024-03-29T11:09:04.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f79d9b640 1 -- 172.21.15.70:0/4248407043 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f64001110 con 0x7f6f7c09ac40 2024-03-29T11:09:04.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f79d9b640 1 -- 172.21.15.70:0/4248407043 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3530329028 0 0) 0x7f6f68003520 con 0x7f6f7c0b4360 2024-03-29T11:09:04.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f79d9b640 1 -- 172.21.15.70:0/4248407043 >> v1:172.21.15.18:6790/0 conn(0x7f6f7c09ac40 legacy=0x7f6f7c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:04.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f79d9b640 1 -- 172.21.15.70:0/4248407043 >> v1:172.21.15.18:6789/0 conn(0x7f6f7c09c870 legacy=0x7f6f7c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:04.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f79d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:09:04.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f79d9b640 1 monclient: found mon.noname-c 2024-03-29T11:09:04.957 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f79d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:04.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f79d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.70:6789/0 2024-03-29T11:09:04.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f79d9b640 1 -- 172.21.15.70:0/4248407043 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f7c0b5160 con 0x7f6f7c0b4360 2024-03-29T11:09:04.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f79d9b640 10 monclient: _finish_auth 0 2024-03-29T11:09:04.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f79d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:09:04.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f79d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:04.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f79d9b640 1 -- 172.21.15.70:0/4248407043 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f6f74001a10 con 0x7f6f7c0b4360 2024-03-29T11:09:04.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f80e4e640 5 monclient: authenticate success, global_id 4117 2024-03-29T11:09:04.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f80e4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:04.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f79d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:04.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f79d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:09:04.958 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f79d9b640 10 monclient: dump: 2024-03-29T11:09:04.958 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:04.958 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:04.958 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:04.959 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:04.959 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:04.959 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:04.959 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:04.959 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:04.959 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:04.959 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:04.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f80e4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:04.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f79d9b640 1 -- 172.21.15.70:0/4248407043 <== mon.1 v1:172.21.15.70:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6f74001bf0 con 0x7f6f7c0b4360 2024-03-29T11:09:04.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f79d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:04.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f79d9b640 1 -- 172.21.15.70:0/4248407043 <== mon.1 v1:172.21.15.70:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f6f74002000 con 0x7f6f7c0b4360 2024-03-29T11:09:04.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f80e4e640 10 monclient: get_monmap_and_config success 2024-03-29T11:09:04.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f80e4e640 4 set_mon_vals no callback set 2024-03-29T11:09:04.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f79d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:04.959 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f79d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:09:04.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.968+0000 7f6f79d9b640 10 monclient: dump: 2024-03-29T11:09:04.960 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:04.960 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:04.960 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:04.960 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:04.960 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:04.960 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:04.960 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:04.960 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:04.960 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:04.960 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:04.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.969+0000 7f6f80e4e640 10 monclient: shutdown 2024-03-29T11:09:04.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.969+0000 7f6f80e4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:04.960 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.969+0000 7f6f80e4e640 1 -- 172.21.15.70:0/4248407043 >> v1:172.21.15.70:6789/0 conn(0x7f6f7c0b4360 legacy=0x7f6f7c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:04.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.969+0000 7f6f80e4e640 1 -- 172.21.15.70:0/4248407043 shutdown_connections 2024-03-29T11:09:04.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.969+0000 7f6f80e4e640 1 -- 172.21.15.70:0/4248407043 >> 172.21.15.70:0/4248407043 conn(0x7f6f7c066b00 msgr2=0x7f6f7c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:04.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.969+0000 7f6f80e4e640 1 -- 172.21.15.70:0/4248407043 shutdown_connections 2024-03-29T11:09:04.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.969+0000 7f6f80e4e640 1 -- 172.21.15.70:0/4248407043 wait complete. 2024-03-29T11:09:04.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.970+0000 7f6f80e4e640 10 monclient: build_initial_monmap 2024-03-29T11:09:04.961 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.970+0000 7f6f80e4e640 10 monclient: monmap: 2024-03-29T11:09:04.961 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-03-29T11:09:04.961 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:09:04.961 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-03-29T11:09:04.961 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-03-29T11:09:04.961 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:04.961 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:04.961 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:09:04.961 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:09:04.962 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:09:04.962 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:04.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.970+0000 7f6f80e4e640 1 librados: starting msgr at 2024-03-29T11:09:04.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.970+0000 7f6f80e4e640 1 librados: starting objecter 2024-03-29T11:09:04.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.970+0000 7f6f80e4e640 1 Processor -- start 2024-03-29T11:09:04.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.970+0000 7f6f80e4e640 1 -- start start 2024-03-29T11:09:04.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.970+0000 7f6f80e4e640 1 librados: setting wanted keys 2024-03-29T11:09:04.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.970+0000 7f6f80e4e640 1 librados: calling monclient init 2024-03-29T11:09:04.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.970+0000 7f6f80e4e640 10 monclient: init 2024-03-29T11:09:04.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f80e4e640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:04.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f80e4e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:09:04.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f80e4e640 10 monclient(hunting): picked mon.noname-b con 0x7f6f7c09ac40 addr v1:172.21.15.70:6789/0 2024-03-29T11:09:04.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f80e4e640 10 monclient(hunting): picked mon.noname-c con 0x7f6f7c1d7cd0 addr v1:172.21.15.18:6790/0 2024-03-29T11:09:04.962 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f80e4e640 10 monclient(hunting): picked mon.noname-a con 0x7f6f7c0ad1e0 addr v1:172.21.15.18:6789/0 2024-03-29T11:09:04.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f80e4e640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f7c1d7130 con 0x7f6f7c0ad1e0 2024-03-29T11:09:04.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f80e4e640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f7c139b10 con 0x7f6f7c09ac40 2024-03-29T11:09:04.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f80e4e640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f7c13ffa0 con 0x7f6f7c1d7cd0 2024-03-29T11:09:04.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f80e4e640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:04.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f80e4e640 10 monclient(hunting): authenticate will time out at 761.199515s 2024-03-29T11:09:04.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f7a59c640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f6f7c09ac40 0x7f6f7c1d75c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:48214/0 (socket says 172.21.15.70:48214) 2024-03-29T11:09:04.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f7a59c640 1 -- 172.21.15.70:0/3214398206 learned_addr learned my addr 172.21.15.70:0/3214398206 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-03-29T11:09:04.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f72ffd640 10 client.?.objecter ms_handle_connect 0x7f6f7c09ac40 2024-03-29T11:09:04.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f72ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:04.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f72ffd640 10 client.?.objecter ms_handle_connect 0x7f6f7c0ad1e0 2024-03-29T11:09:04.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f72ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:04.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f72ffd640 10 client.?.objecter ms_handle_connect 0x7f6f7c1d7cd0 2024-03-29T11:09:04.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f72ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:04.963 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f72ffd640 1 -- 172.21.15.70:0/3214398206 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2427363252 0 0) 0x7f6f7c139b10 con 0x7f6f7c09ac40 2024-03-29T11:09:04.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f72ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:09:04.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:04.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:04.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f72ffd640 10 monclient(hunting): my global_id is 4120 2024-03-29T11:09:04.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f72ffd640 1 -- 172.21.15.70:0/3214398206 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f5c004eb0 con 0x7f6f7c09ac40 2024-03-29T11:09:04.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f72ffd640 1 -- 172.21.15.70:0/3214398206 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1296678350 0 0) 0x7f6f7c1d7130 con 0x7f6f7c0ad1e0 2024-03-29T11:09:04.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f72ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:09:04.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:04.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:04.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f72ffd640 10 monclient(hunting): my global_id is 4122 2024-03-29T11:09:04.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f72ffd640 1 -- 172.21.15.70:0/3214398206 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f7c139b10 con 0x7f6f7c0ad1e0 2024-03-29T11:09:04.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f72ffd640 1 -- 172.21.15.70:0/3214398206 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3960443227 0 0) 0x7f6f7c13ffa0 con 0x7f6f7c1d7cd0 2024-03-29T11:09:04.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f72ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:09:04.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:04.964 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:04.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f72ffd640 10 monclient(hunting): my global_id is 4124 2024-03-29T11:09:04.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f72ffd640 1 -- 172.21.15.70:0/3214398206 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f7c1d7130 con 0x7f6f7c1d7cd0 2024-03-29T11:09:04.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f72ffd640 1 -- 172.21.15.70:0/3214398206 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4280022312 0 0) 0x7f6f5c004eb0 con 0x7f6f7c09ac40 2024-03-29T11:09:04.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f72ffd640 1 -- 172.21.15.70:0/3214398206 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f7c13ffa0 con 0x7f6f7c09ac40 2024-03-29T11:09:04.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f72ffd640 1 -- 172.21.15.70:0/3214398206 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f6f6c001a10 con 0x7f6f7c09ac40 2024-03-29T11:09:04.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f72ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:04.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f72ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:09:04.965 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f72ffd640 10 monclient(hunting): dump: 2024-03-29T11:09:04.965 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:04.965 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:04.965 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:04.965 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:04.965 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:04.965 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:04.965 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:04.966 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:04.966 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:04.966 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:04.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f72ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:09:04.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f72ffd640 1 -- 172.21.15.70:0/3214398206 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4013046966 0 0) 0x7f6f7c139b10 con 0x7f6f7c0ad1e0 2024-03-29T11:09:04.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f72ffd640 1 -- 172.21.15.70:0/3214398206 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f5c004eb0 con 0x7f6f7c0ad1e0 2024-03-29T11:09:04.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f72ffd640 1 -- 172.21.15.70:0/3214398206 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f6f74001f00 con 0x7f6f7c0ad1e0 2024-03-29T11:09:04.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f72ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:04.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f72ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:09:04.966 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f72ffd640 10 monclient(hunting): dump: 2024-03-29T11:09:04.966 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:04.966 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:04.966 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:04.966 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:04.966 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:04.967 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:04.967 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:04.967 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:04.967 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:04.967 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:04.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.971+0000 7f6f72ffd640 1 -- 172.21.15.70:0/3214398206 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 976845557 0 0) 0x7f6f7c1d7130 con 0x7f6f7c1d7cd0 2024-03-29T11:09:04.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.972+0000 7f6f72ffd640 1 -- 172.21.15.70:0/3214398206 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f7c139b10 con 0x7f6f7c1d7cd0 2024-03-29T11:09:04.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.972+0000 7f6f72ffd640 1 -- 172.21.15.70:0/3214398206 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f6f640018a0 con 0x7f6f7c1d7cd0 2024-03-29T11:09:04.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.972+0000 7f6f72ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:04.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.972+0000 7f6f72ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:09:04.967 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.972+0000 7f6f72ffd640 10 monclient(hunting): dump: 2024-03-29T11:09:04.967 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:04.967 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:04.967 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:04.968 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:04.968 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:04.968 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:04.968 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:04.968 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:04.968 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:04.968 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:04.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.972+0000 7f6f72ffd640 1 -- 172.21.15.70:0/3214398206 <== mon.1 v1:172.21.15.70:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4172481383 0 0) 0x7f6f7c13ffa0 con 0x7f6f7c09ac40 2024-03-29T11:09:04.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.972+0000 7f6f72ffd640 1 -- 172.21.15.70:0/3214398206 >> v1:172.21.15.18:6790/0 conn(0x7f6f7c1d7cd0 legacy=0x7f6f7c138cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:04.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.972+0000 7f6f72ffd640 1 -- 172.21.15.70:0/3214398206 >> v1:172.21.15.18:6789/0 conn(0x7f6f7c0ad1e0 legacy=0x7f6f7c139400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:04.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.972+0000 7f6f72ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:09:04.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.972+0000 7f6f72ffd640 1 monclient: found mon.b 2024-03-29T11:09:04.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.972+0000 7f6f72ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:04.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.972+0000 7f6f72ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:04.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.972+0000 7f6f72ffd640 1 -- 172.21.15.70:0/3214398206 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f7c0ae520 con 0x7f6f7c09ac40 2024-03-29T11:09:04.968 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.972+0000 7f6f72ffd640 10 monclient: _finish_auth 0 2024-03-29T11:09:04.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.972+0000 7f6f72ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:09:04.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.972+0000 7f6f72ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:04.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.972+0000 7f6f80e4e640 5 monclient: authenticate success, global_id 4120 2024-03-29T11:09:04.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.972+0000 7f6f80e4e640 10 monclient: _renew_subs 2024-03-29T11:09:04.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.972+0000 7f6f80e4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:04.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.972+0000 7f6f80e4e640 1 -- 172.21.15.70:0/3214398206 --> v1:172.21.15.70:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f7c0ae800 con 0x7f6f7c09ac40 2024-03-29T11:09:04.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.972+0000 7f6f80e4e640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:09:04.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.972+0000 7f6f80e4e640 10 monclient: _renew_subs 2024-03-29T11:09:04.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.972+0000 7f6f80e4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:04.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.972+0000 7f6f80e4e640 1 -- 172.21.15.70:0/3214398206 --> v1:172.21.15.70:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f7c142b70 con 0x7f6f7c09ac40 2024-03-29T11:09:04.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.972+0000 7f6f80e4e640 1 librados: init done 2024-03-29T11:09:04.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.972+0000 7f6f72ffd640 1 -- 172.21.15.70:0/3214398206 <== mon.1 v1:172.21.15.70:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6f6c001cf0 con 0x7f6f7c09ac40 2024-03-29T11:09:04.969 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.972+0000 7f6f72ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:04.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.972+0000 7f6f72ffd640 1 -- 172.21.15.70:0/3214398206 <== mon.1 v1:172.21.15.70:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f6f6c003a60 con 0x7f6f7c09ac40 2024-03-29T11:09:04.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.972+0000 7f6f72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:04.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.972+0000 7f6f72ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:09:04.970 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.972+0000 7f6f72ffd640 10 monclient: dump: 2024-03-29T11:09:04.970 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:04.970 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:04.970 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:04.970 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:04.970 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:04.970 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:04.970 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:04.970 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:04.970 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:04.970 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:04.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.972+0000 7f6f7959a640 4 set_mon_vals no callback set 2024-03-29T11:09:04.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.972+0000 7f6f72ffd640 1 -- 172.21.15.70:0/3214398206 <== mon.1 v1:172.21.15.70:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6f6c003fd0 con 0x7f6f7c09ac40 2024-03-29T11:09:04.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.972+0000 7f6f72ffd640 1 -- 172.21.15.70:0/3214398206 <== mon.1 v1:172.21.15.70:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 2367042358 0 0) 0x7f6f6c006790 con 0x7f6f7c09ac40 2024-03-29T11:09:04.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.972+0000 7f6f72ffd640 10 client.4120.objecter ms_dispatch 0x7f6f7c000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T11:09:04.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.972+0000 7f6f72ffd640 3 client.4120.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T11:09:04.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.972+0000 7f6f72ffd640 3 client.4120.objecter handle_osd_map decoding full epoch 3 2024-03-29T11:09:04.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.972+0000 7f6f72ffd640 20 client.4120.objecter dump_active .. 0 homeless 2024-03-29T11:09:04.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.973+0000 7f6f80e4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:04.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.973+0000 7f6f80e4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:04.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.973+0000 7f6f80e4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:04.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.973+0000 7f6f80e4e640 1 -- 172.21.15.70:0/3214398206 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f60005080 con 0x7f6f7c09ac40 2024-03-29T11:09:04.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.976+0000 7f6f72ffd640 1 -- 172.21.15.70:0/3214398206 <== mon.1 v1:172.21.15.70:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6f6c0069f0 con 0x7f6f7c09ac40 2024-03-29T11:09:04.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.976+0000 7f6f72ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:04.971 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:04.976+0000 7f6f72ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:09:05.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.048+0000 7f6f80e4e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4a8fc337-370b-4c2a-83d7-d5b6e9fb2436", "id": 1}] 2024-03-29T11:09:05.026 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.048+0000 7f6f80e4e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4a8fc337-370b-4c2a-83d7-d5b6e9fb2436", "id": 1}] 2024-03-29T11:09:05.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.048+0000 7f6f80e4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:05.027 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.048+0000 7f6f80e4e640 1 -- 172.21.15.70:0/3214398206 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "osd new", "uuid": "4a8fc337-370b-4c2a-83d7-d5b6e9fb2436", "id": 1} v 0) v1 -- 0x7f6f600058b0 con 0x7f6f7c09ac40 2024-03-29T11:09:05.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.050+0000 7f6f72ffd640 1 -- 172.21.15.70:0/3214398206 <== mon.1 v1:172.21.15.70:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4a8fc337-370b-4c2a-83d7-d5b6e9fb2436", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 1533231761 0 940483276) 0x7f6f6c006b90 con 0x7f6f7c09ac40 2024-03-29T11:09:05.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.050+0000 7f6f72ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4a8fc337-370b-4c2a-83d7-d5b6e9fb2436", "id": 1}] 2024-03-29T11:09:05.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.050+0000 7f6f72ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:09:05.029 INFO:teuthology.orchestra.run.smithi070.stdout:1 2024-03-29T11:09:05.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.051+0000 7f6f80e4e640 10 librados: watch_flush enter 2024-03-29T11:09:05.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.051+0000 7f6f80e4e640 10 librados: watch_flush exit 2024-03-29T11:09:05.030 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.051+0000 7f6f80e4e640 20 client.4120.objecter shutdown clearing up homeless session... 2024-03-29T11:09:05.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.051+0000 7f6f80e4e640 10 client.4120.objecter successfully canceled tick 2024-03-29T11:09:05.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.051+0000 7f6f80e4e640 10 monclient: shutdown 2024-03-29T11:09:05.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.051+0000 7f6f80e4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:05.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.051+0000 7f6f80e4e640 1 -- 172.21.15.70:0/3214398206 >> v1:172.21.15.70:6789/0 conn(0x7f6f7c09ac40 legacy=0x7f6f7c1d75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:05.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.051+0000 7f6f80e4e640 1 -- 172.21.15.70:0/3214398206 shutdown_connections 2024-03-29T11:09:05.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.051+0000 7f6f80e4e640 1 -- 172.21.15.70:0/3214398206 >> 172.21.15.70:0/3214398206 conn(0x7f6f7c066b00 msgr2=0x7f6f7c0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:05.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.051+0000 7f6f80e4e640 1 -- 172.21.15.70:0/3214398206 shutdown_connections 2024-03-29T11:09:05.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.051+0000 7f6f80e4e640 1 -- 172.21.15.70:0/3214398206 wait complete. 2024-03-29T11:09:05.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.051+0000 7f6f80e4e640 1 librados: shutdown 2024-03-29T11:09:05.040 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --cluster ceph osd new 9e0ff187-0cec-457f-9b8b-2c925ac96306 2 2024-03-29T11:09:05.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.160+0000 7fdb62f39640 10 monclient: get_monmap_and_config 2024-03-29T11:09:05.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.160+0000 7fdb62f39640 10 monclient: build_initial_monmap 2024-03-29T11:09:05.139 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.160+0000 7fdb62f39640 10 monclient: monmap: 2024-03-29T11:09:05.139 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-03-29T11:09:05.139 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:05.139 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:09:05.161703+0000 2024-03-29T11:09:05.139 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:09:05.161703+0000 2024-03-29T11:09:05.139 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:05.139 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:05.139 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:09:05.140 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:09:05.140 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:09:05.140 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:05.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.161+0000 7fdb62f39640 1 Processor -- start 2024-03-29T11:09:05.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.161+0000 7fdb62f39640 1 -- start start 2024-03-29T11:09:05.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.161+0000 7fdb62f39640 10 monclient: init 2024-03-29T11:09:05.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.161+0000 7fdb62f39640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:05.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.161+0000 7fdb62f39640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:09:05.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb62f39640 10 monclient(hunting): picked mon.noname-b con 0x7fdb5c09ac40 addr v1:172.21.15.18:6790/0 2024-03-29T11:09:05.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb62f39640 10 monclient(hunting): picked mon.noname-c con 0x7fdb5c09c870 addr v1:172.21.15.70:6789/0 2024-03-29T11:09:05.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb62f39640 10 monclient(hunting): picked mon.noname-a con 0x7fdb5c0b4360 addr v1:172.21.15.18:6789/0 2024-03-29T11:09:05.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb62f39640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb5c09dec0 con 0x7fdb5c0b4360 2024-03-29T11:09:05.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb62f39640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb5c09da30 con 0x7fdb5c09c870 2024-03-29T11:09:05.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb62f39640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb5c0b39c0 con 0x7fdb5c09ac40 2024-03-29T11:09:05.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb62f39640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:05.148 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb62f39640 10 monclient(hunting): authenticate will time out at 761.390262s 2024-03-29T11:09:05.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5bfff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fdb5c09c870 0x7fdb5c13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:48228/0 (socket says 172.21.15.70:48228) 2024-03-29T11:09:05.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5bfff640 1 -- 172.21.15.70:0/594659559 learned_addr learned my addr 172.21.15.70:0/594659559 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-03-29T11:09:05.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5b7fe640 1 -- 172.21.15.70:0/594659559 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1919733651 0 0) 0x7fdb5c09da30 con 0x7fdb5c09c870 2024-03-29T11:09:05.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:09:05.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:05.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:05.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5b7fe640 10 monclient(hunting): my global_id is 4123 2024-03-29T11:09:05.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5b7fe640 1 -- 172.21.15.70:0/594659559 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb4c003520 con 0x7fdb5c09c870 2024-03-29T11:09:05.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5b7fe640 1 -- 172.21.15.70:0/594659559 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 201778033 0 0) 0x7fdb54001490 con 0x7fdb5c0b4360 2024-03-29T11:09:05.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:09:05.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:05.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:05.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5b7fe640 10 monclient(hunting): my global_id is 4125 2024-03-29T11:09:05.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5b7fe640 1 -- 172.21.15.70:0/594659559 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb5c09da30 con 0x7fdb5c0b4360 2024-03-29T11:09:05.149 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5b7fe640 1 -- 172.21.15.70:0/594659559 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1515024591 0 0) 0x7fdb5c0b39c0 con 0x7fdb5c09ac40 2024-03-29T11:09:05.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:09:05.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:05.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:05.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5b7fe640 10 monclient(hunting): my global_id is 4127 2024-03-29T11:09:05.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5b7fe640 1 -- 172.21.15.70:0/594659559 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb54001490 con 0x7fdb5c09ac40 2024-03-29T11:09:05.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5b7fe640 1 -- 172.21.15.70:0/594659559 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 563746080 0 0) 0x7fdb4c003520 con 0x7fdb5c09c870 2024-03-29T11:09:05.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5b7fe640 1 -- 172.21.15.70:0/594659559 >> v1:172.21.15.18:6790/0 conn(0x7fdb5c09ac40 legacy=0x7fdb5c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:05.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5b7fe640 1 -- 172.21.15.70:0/594659559 >> v1:172.21.15.18:6789/0 conn(0x7fdb5c0b4360 legacy=0x7fdb5c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:05.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:09:05.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5b7fe640 1 monclient: found mon.noname-c 2024-03-29T11:09:05.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:05.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.70:6789/0 2024-03-29T11:09:05.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5b7fe640 1 -- 172.21.15.70:0/594659559 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb5c0b5160 con 0x7fdb5c09c870 2024-03-29T11:09:05.150 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:09:05.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:09:05.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:05.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5b7fe640 1 -- 172.21.15.70:0/594659559 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fdb48001690 con 0x7fdb5c09c870 2024-03-29T11:09:05.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb62f39640 5 monclient: authenticate success, global_id 4123 2024-03-29T11:09:05.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb62f39640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:05.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:05.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:09:05.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5b7fe640 10 monclient: dump: 2024-03-29T11:09:05.151 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:05.151 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:05.151 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.151 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.151 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:05.151 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:05.152 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:05.152 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:05.152 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:05.152 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:05.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb62f39640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:05.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5b7fe640 1 -- 172.21.15.70:0/594659559 <== mon.1 v1:172.21.15.70:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb48001870 con 0x7fdb5c09c870 2024-03-29T11:09:05.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:05.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5b7fe640 1 -- 172.21.15.70:0/594659559 <== mon.1 v1:172.21.15.70:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fdb48001c80 con 0x7fdb5c09c870 2024-03-29T11:09:05.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb62f39640 10 monclient: get_monmap_and_config success 2024-03-29T11:09:05.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb62f39640 4 set_mon_vals no callback set 2024-03-29T11:09:05.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:05.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:09:05.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.162+0000 7fdb5b7fe640 10 monclient: dump: 2024-03-29T11:09:05.152 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:05.152 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:05.153 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.153 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.153 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:05.153 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:05.153 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:05.153 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:05.153 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:05.153 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:05.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.163+0000 7fdb62f39640 10 monclient: shutdown 2024-03-29T11:09:05.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.163+0000 7fdb62f39640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:05.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.163+0000 7fdb62f39640 1 -- 172.21.15.70:0/594659559 >> v1:172.21.15.70:6789/0 conn(0x7fdb5c09c870 legacy=0x7fdb5c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:05.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.163+0000 7fdb62f39640 1 -- 172.21.15.70:0/594659559 shutdown_connections 2024-03-29T11:09:05.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.163+0000 7fdb62f39640 1 -- 172.21.15.70:0/594659559 >> 172.21.15.70:0/594659559 conn(0x7fdb5c066b00 msgr2=0x7fdb5c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:05.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.163+0000 7fdb62f39640 1 -- 172.21.15.70:0/594659559 shutdown_connections 2024-03-29T11:09:05.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.163+0000 7fdb62f39640 1 -- 172.21.15.70:0/594659559 wait complete. 2024-03-29T11:09:05.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.164+0000 7fdb62f39640 10 monclient: build_initial_monmap 2024-03-29T11:09:05.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.164+0000 7fdb62f39640 10 monclient: monmap: 2024-03-29T11:09:05.154 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-03-29T11:09:05.154 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:09:05.154 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-03-29T11:09:05.154 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-03-29T11:09:05.154 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:05.154 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:05.154 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:09:05.154 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:09:05.154 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:09:05.154 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:05.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.164+0000 7fdb62f39640 1 librados: starting msgr at 2024-03-29T11:09:05.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.164+0000 7fdb62f39640 1 librados: starting objecter 2024-03-29T11:09:05.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.164+0000 7fdb62f39640 1 Processor -- start 2024-03-29T11:09:05.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.164+0000 7fdb62f39640 1 -- start start 2024-03-29T11:09:05.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.164+0000 7fdb62f39640 1 librados: setting wanted keys 2024-03-29T11:09:05.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.164+0000 7fdb62f39640 1 librados: calling monclient init 2024-03-29T11:09:05.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.164+0000 7fdb62f39640 10 monclient: init 2024-03-29T11:09:05.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb62f39640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:05.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb62f39640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:09:05.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb62f39640 10 monclient(hunting): picked mon.noname-c con 0x7fdb5c09ac40 addr v1:172.21.15.18:6790/0 2024-03-29T11:09:05.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb62f39640 10 monclient(hunting): picked mon.noname-b con 0x7fdb5c0b6b00 addr v1:172.21.15.70:6789/0 2024-03-29T11:09:05.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb62f39640 10 monclient(hunting): picked mon.noname-a con 0x7fdb5c1406b0 addr v1:172.21.15.18:6789/0 2024-03-29T11:09:05.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb62f39640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb5c1d7e20 con 0x7fdb5c1406b0 2024-03-29T11:09:05.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb62f39640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb5c140b00 con 0x7fdb5c0b6b00 2024-03-29T11:09:05.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb62f39640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb5c0ad080 con 0x7fdb5c09ac40 2024-03-29T11:09:05.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb62f39640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:05.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb62f39640 10 monclient(hunting): authenticate will time out at 761.393320s 2024-03-29T11:09:05.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb5bfff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fdb5c0b6b00 0x7fdb5c13ffa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:48232/0 (socket says 172.21.15.70:48232) 2024-03-29T11:09:05.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb5bfff640 1 -- 172.21.15.70:0/1405020635 learned_addr learned my addr 172.21.15.70:0/1405020635 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-03-29T11:09:05.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 10 client.?.objecter ms_handle_connect 0x7fdb5c0b6b00 2024-03-29T11:09:05.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:05.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 10 client.?.objecter ms_handle_connect 0x7fdb5c09ac40 2024-03-29T11:09:05.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:05.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 10 client.?.objecter ms_handle_connect 0x7fdb5c1406b0 2024-03-29T11:09:05.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:05.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 1 -- 172.21.15.70:0/1405020635 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2644500719 0 0) 0x7fdb5c140b00 con 0x7fdb5c0b6b00 2024-03-29T11:09:05.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 10 monclient(hunting): _negotiate 2024-03-29T11:09:05.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:05.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:05.156 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 10 monclient(hunting): my global_id is 4126 2024-03-29T11:09:05.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 1 -- 172.21.15.70:0/1405020635 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb40004eb0 con 0x7fdb5c0b6b00 2024-03-29T11:09:05.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 1 -- 172.21.15.70:0/1405020635 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 468473649 0 0) 0x7fdb5c1d7e20 con 0x7fdb5c1406b0 2024-03-29T11:09:05.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 10 monclient(hunting): _negotiate 2024-03-29T11:09:05.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:05.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:05.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 10 monclient(hunting): my global_id is 4128 2024-03-29T11:09:05.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 1 -- 172.21.15.70:0/1405020635 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb5c140b00 con 0x7fdb5c1406b0 2024-03-29T11:09:05.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 1 -- 172.21.15.70:0/1405020635 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1499155604 0 0) 0x7fdb5c0ad080 con 0x7fdb5c09ac40 2024-03-29T11:09:05.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 10 monclient(hunting): _negotiate 2024-03-29T11:09:05.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:05.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:05.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 10 monclient(hunting): my global_id is 4130 2024-03-29T11:09:05.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 1 -- 172.21.15.70:0/1405020635 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb5c1d7e20 con 0x7fdb5c09ac40 2024-03-29T11:09:05.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 1 -- 172.21.15.70:0/1405020635 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1405296312 0 0) 0x7fdb40004eb0 con 0x7fdb5c0b6b00 2024-03-29T11:09:05.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 1 -- 172.21.15.70:0/1405020635 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb5c0ad080 con 0x7fdb5c0b6b00 2024-03-29T11:09:05.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 1 -- 172.21.15.70:0/1405020635 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fdb48001bc0 con 0x7fdb5c0b6b00 2024-03-29T11:09:05.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:05.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:09:05.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 10 monclient(hunting): dump: 2024-03-29T11:09:05.158 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:05.158 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:05.158 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.158 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.158 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:05.158 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:05.158 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:05.158 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:05.158 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:05.158 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:05.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:09:05.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 1 -- 172.21.15.70:0/1405020635 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1906800593 0 0) 0x7fdb5c140b00 con 0x7fdb5c1406b0 2024-03-29T11:09:05.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 1 -- 172.21.15.70:0/1405020635 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb40004eb0 con 0x7fdb5c1406b0 2024-03-29T11:09:05.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 1 -- 172.21.15.70:0/1405020635 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3641491218 0 0) 0x7fdb5c1d7e20 con 0x7fdb5c09ac40 2024-03-29T11:09:05.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 1 -- 172.21.15.70:0/1405020635 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb5c140b00 con 0x7fdb5c09ac40 2024-03-29T11:09:05.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 1 -- 172.21.15.70:0/1405020635 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fdb54001b40 con 0x7fdb5c1406b0 2024-03-29T11:09:05.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:05.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:09:05.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 10 monclient(hunting): dump: 2024-03-29T11:09:05.159 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:05.159 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:05.159 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.159 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.159 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:05.160 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:05.160 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:05.160 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:05.160 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:05.160 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:05.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 1 -- 172.21.15.70:0/1405020635 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fdb500016e0 con 0x7fdb5c09ac40 2024-03-29T11:09:05.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:05.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:09:05.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 10 monclient(hunting): dump: 2024-03-29T11:09:05.160 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:05.160 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:05.160 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.160 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.160 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:05.161 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:05.161 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:05.161 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:05.161 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:05.161 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:05.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 1 -- 172.21.15.70:0/1405020635 <== mon.1 v1:172.21.15.70:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3882479141 0 0) 0x7fdb5c0ad080 con 0x7fdb5c0b6b00 2024-03-29T11:09:05.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 1 -- 172.21.15.70:0/1405020635 >> v1:172.21.15.18:6790/0 conn(0x7fdb5c09ac40 legacy=0x7fdb5c1d73a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:05.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 1 -- 172.21.15.70:0/1405020635 >> v1:172.21.15.18:6789/0 conn(0x7fdb5c1406b0 legacy=0x7fdb5c1e1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:05.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 10 monclient: _finish_hunting 0 2024-03-29T11:09:05.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 1 monclient: found mon.b 2024-03-29T11:09:05.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:05.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:05.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 1 -- 172.21.15.70:0/1405020635 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb5c1d7ab0 con 0x7fdb5c0b6b00 2024-03-29T11:09:05.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 10 monclient: _finish_auth 0 2024-03-29T11:09:05.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 10 monclient: _check_auth_tickets 2024-03-29T11:09:05.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.165+0000 7fdb597fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:05.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.166+0000 7fdb62f39640 5 monclient: authenticate success, global_id 4126 2024-03-29T11:09:05.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.166+0000 7fdb62f39640 10 monclient: _renew_subs 2024-03-29T11:09:05.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.166+0000 7fdb62f39640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:05.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.166+0000 7fdb62f39640 1 -- 172.21.15.70:0/1405020635 --> v1:172.21.15.70:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb5c1e3570 con 0x7fdb5c0b6b00 2024-03-29T11:09:05.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.166+0000 7fdb62f39640 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:09:05.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.166+0000 7fdb62f39640 10 monclient: _renew_subs 2024-03-29T11:09:05.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.166+0000 7fdb62f39640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:05.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.166+0000 7fdb62f39640 1 -- 172.21.15.70:0/1405020635 --> v1:172.21.15.70:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb5c1e38e0 con 0x7fdb5c0b6b00 2024-03-29T11:09:05.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.166+0000 7fdb62f39640 1 librados: init done 2024-03-29T11:09:05.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.166+0000 7fdb597fa640 1 -- 172.21.15.70:0/1405020635 <== mon.1 v1:172.21.15.70:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb48004bf0 con 0x7fdb5c0b6b00 2024-03-29T11:09:05.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.166+0000 7fdb597fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:05.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.166+0000 7fdb597fa640 1 -- 172.21.15.70:0/1405020635 <== mon.1 v1:172.21.15.70:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fdb48004f00 con 0x7fdb5c0b6b00 2024-03-29T11:09:05.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.166+0000 7fdb597fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:05.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.166+0000 7fdb597fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:09:05.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.166+0000 7fdb5affd640 4 set_mon_vals no callback set 2024-03-29T11:09:05.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.166+0000 7fdb597fa640 10 monclient: dump: 2024-03-29T11:09:05.163 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:05.163 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:05.163 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.163 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.163 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:05.163 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:05.163 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:05.163 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:05.163 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:05.163 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:05.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.166+0000 7fdb597fa640 1 -- 172.21.15.70:0/1405020635 <== mon.1 v1:172.21.15.70:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fdb48005470 con 0x7fdb5c0b6b00 2024-03-29T11:09:05.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.166+0000 7fdb597fa640 1 -- 172.21.15.70:0/1405020635 <== mon.1 v1:172.21.15.70:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 1170227196 0 0) 0x7fdb48005b40 con 0x7fdb5c0b6b00 2024-03-29T11:09:05.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.166+0000 7fdb597fa640 10 client.4126.objecter ms_dispatch 0x7fdb5c000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T11:09:05.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.166+0000 7fdb597fa640 3 client.4126.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T11:09:05.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.166+0000 7fdb597fa640 3 client.4126.objecter handle_osd_map decoding full epoch 4 2024-03-29T11:09:05.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.166+0000 7fdb597fa640 20 client.4126.objecter dump_active .. 0 homeless 2024-03-29T11:09:05.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.166+0000 7fdb62f39640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:05.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.166+0000 7fdb62f39640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:05.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.166+0000 7fdb62f39640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:05.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.166+0000 7fdb62f39640 1 -- 172.21.15.70:0/1405020635 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb3c005080 con 0x7fdb5c0b6b00 2024-03-29T11:09:05.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.170+0000 7fdb597fa640 1 -- 172.21.15.70:0/1405020635 <== mon.1 v1:172.21.15.70:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fdb48005720 con 0x7fdb5c0b6b00 2024-03-29T11:09:05.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.170+0000 7fdb597fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:05.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.170+0000 7fdb597fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:09:05.192 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-29T11:09:05.214+0000 7fad3de2e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:09:05.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.240+0000 7fdb62f39640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9e0ff187-0cec-457f-9b8b-2c925ac96306", "id": 2}] 2024-03-29T11:09:05.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.240+0000 7fdb62f39640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9e0ff187-0cec-457f-9b8b-2c925ac96306", "id": 2}] 2024-03-29T11:09:05.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.240+0000 7fdb62f39640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:05.220 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.240+0000 7fdb62f39640 1 -- 172.21.15.70:0/1405020635 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "osd new", "uuid": "9e0ff187-0cec-457f-9b8b-2c925ac96306", "id": 2} v 0) v1 -- 0x7fdb3c0058b0 con 0x7fdb5c0b6b00 2024-03-29T11:09:05.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.244+0000 7fdb597fa640 1 -- 172.21.15.70:0/1405020635 <== mon.1 v1:172.21.15.70:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9e0ff187-0cec-457f-9b8b-2c925ac96306", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 3535736221 0 216599125) 0x7fdb48005da0 con 0x7fdb5c0b6b00 2024-03-29T11:09:05.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.244+0000 7fdb597fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9e0ff187-0cec-457f-9b8b-2c925ac96306", "id": 2}] 2024-03-29T11:09:05.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.244+0000 7fdb597fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:09:05.223 INFO:teuthology.orchestra.run.smithi070.stdout:2 2024-03-29T11:09:05.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.245+0000 7fdb62f39640 10 librados: watch_flush enter 2024-03-29T11:09:05.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.245+0000 7fdb62f39640 10 librados: watch_flush exit 2024-03-29T11:09:05.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.245+0000 7fdb62f39640 20 client.4126.objecter shutdown clearing up homeless session... 2024-03-29T11:09:05.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.245+0000 7fdb62f39640 10 client.4126.objecter successfully canceled tick 2024-03-29T11:09:05.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.245+0000 7fdb62f39640 10 monclient: shutdown 2024-03-29T11:09:05.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.245+0000 7fdb62f39640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:05.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.245+0000 7fdb62f39640 1 -- 172.21.15.70:0/1405020635 >> v1:172.21.15.70:6789/0 conn(0x7fdb5c0b6b00 legacy=0x7fdb5c13ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:05.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.245+0000 7fdb62f39640 1 -- 172.21.15.70:0/1405020635 shutdown_connections 2024-03-29T11:09:05.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.245+0000 7fdb62f39640 1 -- 172.21.15.70:0/1405020635 >> 172.21.15.70:0/1405020635 conn(0x7fdb5c066b00 msgr2=0x7fdb5c0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:05.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.245+0000 7fdb62f39640 1 -- 172.21.15.70:0/1405020635 shutdown_connections 2024-03-29T11:09:05.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.245+0000 7fdb62f39640 1 -- 172.21.15.70:0/1405020635 wait complete. 2024-03-29T11:09:05.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.245+0000 7fdb62f39640 1 librados: shutdown 2024-03-29T11:09:05.228 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-03-29T11:09:05.249+0000 7f6963a2c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:09:05.234 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --cluster ceph osd new c74142fb-d4f2-4e57-a784-70d935727b13 3 2024-03-29T11:09:05.249 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-29T11:09:05.271+0000 7fad3de2e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:09:05.286 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-03-29T11:09:05.307+0000 7f6963a2c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:09:05.295 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-29T11:09:05.317+0000 7fad3de2e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:09:05.332 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-03-29T11:09:05.354+0000 7f6963a2c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:09:05.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.355+0000 7fd8850fb640 10 monclient: get_monmap_and_config 2024-03-29T11:09:05.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.355+0000 7fd8850fb640 10 monclient: build_initial_monmap 2024-03-29T11:09:05.333 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.355+0000 7fd8850fb640 10 monclient: monmap: 2024-03-29T11:09:05.333 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-03-29T11:09:05.333 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:05.334 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:09:05.356161+0000 2024-03-29T11:09:05.334 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:09:05.356161+0000 2024-03-29T11:09:05.334 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:05.334 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:05.334 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:09:05.334 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:09:05.334 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:09:05.334 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:05.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.355+0000 7fd8850fb640 1 Processor -- start 2024-03-29T11:09:05.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.355+0000 7fd8850fb640 1 -- start start 2024-03-29T11:09:05.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.355+0000 7fd8850fb640 10 monclient: init 2024-03-29T11:09:05.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd8850fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:05.343 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd8850fb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:09:05.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd8850fb640 10 monclient(hunting): picked mon.noname-a con 0x7fd8800b1780 addr v1:172.21.15.18:6789/0 2024-03-29T11:09:05.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd8850fb640 10 monclient(hunting): picked mon.noname-c con 0x7fd88013d7e0 addr v1:172.21.15.70:6789/0 2024-03-29T11:09:05.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd8850fb640 10 monclient(hunting): picked mon.noname-b con 0x7fd880143be0 addr v1:172.21.15.18:6790/0 2024-03-29T11:09:05.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd8850fb640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd880070b30 con 0x7fd8800b1780 2024-03-29T11:09:05.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd8850fb640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd880070da0 con 0x7fd88013d7e0 2024-03-29T11:09:05.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd8850fb640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd88009adb0 con 0x7fd880143be0 2024-03-29T11:09:05.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd8850fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:05.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd8850fb640 10 monclient(hunting): authenticate will time out at 761.584531s 2024-03-29T11:09:05.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd87e59c640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fd88013d7e0 0x7fd8801434d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:48240/0 (socket says 172.21.15.70:48240) 2024-03-29T11:09:05.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd87e59c640 1 -- 172.21.15.70:0/2740625218 learned_addr learned my addr 172.21.15.70:0/2740625218 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-03-29T11:09:05.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd87dd9b640 1 -- 172.21.15.70:0/2740625218 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1596395701 0 0) 0x7fd880070da0 con 0x7fd88013d7e0 2024-03-29T11:09:05.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd87dd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:09:05.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd87dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:05.344 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd87dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:05.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd87dd9b640 10 monclient(hunting): my global_id is 4129 2024-03-29T11:09:05.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd87dd9b640 1 -- 172.21.15.70:0/2740625218 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd868003520 con 0x7fd88013d7e0 2024-03-29T11:09:05.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd87dd9b640 1 -- 172.21.15.70:0/2740625218 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1450598423 0 0) 0x7fd88009adb0 con 0x7fd880143be0 2024-03-29T11:09:05.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd87dd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:09:05.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd87dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:05.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd87dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:05.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd87dd9b640 10 monclient(hunting): my global_id is 4133 2024-03-29T11:09:05.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd87dd9b640 1 -- 172.21.15.70:0/2740625218 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd880070da0 con 0x7fd880143be0 2024-03-29T11:09:05.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd87dd9b640 1 -- 172.21.15.70:0/2740625218 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 9488241 0 0) 0x7fd880070b30 con 0x7fd8800b1780 2024-03-29T11:09:05.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd87dd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:09:05.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd87dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:05.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd87dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:05.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd87dd9b640 10 monclient(hunting): my global_id is 4131 2024-03-29T11:09:05.345 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd87dd9b640 1 -- 172.21.15.70:0/2740625218 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd88009adb0 con 0x7fd8800b1780 2024-03-29T11:09:05.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd87dd9b640 1 -- 172.21.15.70:0/2740625218 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1469709889 0 0) 0x7fd868003520 con 0x7fd88013d7e0 2024-03-29T11:09:05.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd87dd9b640 1 -- 172.21.15.70:0/2740625218 >> v1:172.21.15.18:6790/0 conn(0x7fd880143be0 legacy=0x7fd88014b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:05.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd87dd9b640 1 -- 172.21.15.70:0/2740625218 >> v1:172.21.15.18:6789/0 conn(0x7fd8800b1780 legacy=0x7fd88013d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:05.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd87dd9b640 10 monclient: _finish_hunting 0 2024-03-29T11:09:05.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd87dd9b640 1 monclient: found mon.noname-c 2024-03-29T11:09:05.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd87dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:05.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd87dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.70:6789/0 2024-03-29T11:09:05.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd87dd9b640 1 -- 172.21.15.70:0/2740625218 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd880144070 con 0x7fd88013d7e0 2024-03-29T11:09:05.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd87dd9b640 10 monclient: _finish_auth 0 2024-03-29T11:09:05.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd87dd9b640 10 monclient: _check_auth_tickets 2024-03-29T11:09:05.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd87dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:05.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd8850fb640 5 monclient: authenticate success, global_id 4129 2024-03-29T11:09:05.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd87dd9b640 1 -- 172.21.15.70:0/2740625218 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fd870001690 con 0x7fd88013d7e0 2024-03-29T11:09:05.346 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd87dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:05.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd87dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:09:05.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd87dd9b640 10 monclient: dump: 2024-03-29T11:09:05.347 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:05.347 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:05.347 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.347 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.347 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:05.347 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:05.347 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:05.347 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:05.347 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:05.347 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:05.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.356+0000 7fd8850fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:05.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.357+0000 7fd87dd9b640 1 -- 172.21.15.70:0/2740625218 <== mon.1 v1:172.21.15.70:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8700017f0 con 0x7fd88013d7e0 2024-03-29T11:09:05.347 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.357+0000 7fd87dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:05.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.357+0000 7fd8850fb640 10 monclient: get_monmap_and_config success 2024-03-29T11:09:05.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.357+0000 7fd87dd9b640 1 -- 172.21.15.70:0/2740625218 <== mon.1 v1:172.21.15.70:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fd870001c00 con 0x7fd88013d7e0 2024-03-29T11:09:05.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.357+0000 7fd87dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:05.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.357+0000 7fd8850fb640 4 set_mon_vals no callback set 2024-03-29T11:09:05.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.357+0000 7fd87dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:09:05.348 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.357+0000 7fd87dd9b640 10 monclient: dump: 2024-03-29T11:09:05.348 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:05.348 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:05.348 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.348 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.348 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:05.348 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:05.348 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:05.348 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:05.348 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:05.349 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:05.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.357+0000 7fd8850fb640 10 monclient: shutdown 2024-03-29T11:09:05.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.357+0000 7fd8850fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:05.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.357+0000 7fd8850fb640 1 -- 172.21.15.70:0/2740625218 >> v1:172.21.15.70:6789/0 conn(0x7fd88013d7e0 legacy=0x7fd8801434d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:05.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.357+0000 7fd8850fb640 1 -- 172.21.15.70:0/2740625218 shutdown_connections 2024-03-29T11:09:05.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.357+0000 7fd8850fb640 1 -- 172.21.15.70:0/2740625218 >> 172.21.15.70:0/2740625218 conn(0x7fd8800b58f0 msgr2=0x7fd8800b5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:05.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.357+0000 7fd8850fb640 1 -- 172.21.15.70:0/2740625218 shutdown_connections 2024-03-29T11:09:05.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.357+0000 7fd8850fb640 1 -- 172.21.15.70:0/2740625218 wait complete. 2024-03-29T11:09:05.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.358+0000 7fd8850fb640 10 monclient: build_initial_monmap 2024-03-29T11:09:05.349 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.358+0000 7fd8850fb640 10 monclient: monmap: 2024-03-29T11:09:05.349 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-03-29T11:09:05.349 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:09:05.349 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-03-29T11:09:05.349 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-03-29T11:09:05.350 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:05.350 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:05.350 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:09:05.350 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:09:05.350 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:09:05.350 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:05.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.358+0000 7fd8850fb640 1 librados: starting msgr at 2024-03-29T11:09:05.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.358+0000 7fd8850fb640 1 librados: starting objecter 2024-03-29T11:09:05.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.358+0000 7fd8850fb640 1 Processor -- start 2024-03-29T11:09:05.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.358+0000 7fd8850fb640 1 -- start start 2024-03-29T11:09:05.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.358+0000 7fd8850fb640 1 librados: setting wanted keys 2024-03-29T11:09:05.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.358+0000 7fd8850fb640 1 librados: calling monclient init 2024-03-29T11:09:05.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.358+0000 7fd8850fb640 10 monclient: init 2024-03-29T11:09:05.350 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.359+0000 7fd8850fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:05.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.359+0000 7fd8850fb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:09:05.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.359+0000 7fd8850fb640 10 monclient(hunting): picked mon.noname-b con 0x7fd8800b1780 addr v1:172.21.15.70:6789/0 2024-03-29T11:09:05.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.359+0000 7fd8850fb640 10 monclient(hunting): picked mon.noname-c con 0x7fd88013d7e0 addr v1:172.21.15.18:6790/0 2024-03-29T11:09:05.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.359+0000 7fd8850fb640 10 monclient(hunting): picked mon.noname-a con 0x7fd880143be0 addr v1:172.21.15.18:6789/0 2024-03-29T11:09:05.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.359+0000 7fd8850fb640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd88013a010 con 0x7fd880143be0 2024-03-29T11:09:05.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.359+0000 7fd8850fb640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd88013a2f0 con 0x7fd8800b1780 2024-03-29T11:09:05.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.359+0000 7fd8850fb640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd88013a5d0 con 0x7fd88013d7e0 2024-03-29T11:09:05.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.359+0000 7fd8850fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:05.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.359+0000 7fd8850fb640 10 monclient(hunting): authenticate will time out at 761.587702s 2024-03-29T11:09:05.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.359+0000 7fd87ed9d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fd8800b1780 0x7fd8801dae60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:48254/0 (socket says 172.21.15.70:48254) 2024-03-29T11:09:05.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.359+0000 7fd87ed9d640 1 -- 172.21.15.70:0/3427184638 learned_addr learned my addr 172.21.15.70:0/3427184638 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-03-29T11:09:05.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.359+0000 7fd8677fe640 10 client.?.objecter ms_handle_connect 0x7fd8800b1780 2024-03-29T11:09:05.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.359+0000 7fd8677fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:05.351 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.359+0000 7fd8677fe640 10 client.?.objecter ms_handle_connect 0x7fd880143be0 2024-03-29T11:09:05.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.359+0000 7fd8677fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:05.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.359+0000 7fd8677fe640 10 client.?.objecter ms_handle_connect 0x7fd88013d7e0 2024-03-29T11:09:05.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.359+0000 7fd8677fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:05.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.359+0000 7fd8677fe640 1 -- 172.21.15.70:0/3427184638 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 532817591 0 0) 0x7fd88013a2f0 con 0x7fd8800b1780 2024-03-29T11:09:05.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.359+0000 7fd8677fe640 10 monclient(hunting): _negotiate 2024-03-29T11:09:05.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.359+0000 7fd8677fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:05.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.359+0000 7fd8677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:05.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.359+0000 7fd8677fe640 10 monclient(hunting): my global_id is 4132 2024-03-29T11:09:05.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.359+0000 7fd8677fe640 1 -- 172.21.15.70:0/3427184638 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd86c004eb0 con 0x7fd8800b1780 2024-03-29T11:09:05.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.359+0000 7fd8677fe640 1 -- 172.21.15.70:0/3427184638 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1764228950 0 0) 0x7fd88013a5d0 con 0x7fd88013d7e0 2024-03-29T11:09:05.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.359+0000 7fd8677fe640 10 monclient(hunting): _negotiate 2024-03-29T11:09:05.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.359+0000 7fd8677fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:05.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.359+0000 7fd8677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:05.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.359+0000 7fd8677fe640 10 monclient(hunting): my global_id is 4136 2024-03-29T11:09:05.352 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.359+0000 7fd8677fe640 1 -- 172.21.15.70:0/3427184638 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd88013a2f0 con 0x7fd88013d7e0 2024-03-29T11:09:05.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.359+0000 7fd8677fe640 1 -- 172.21.15.70:0/3427184638 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 828516411 0 0) 0x7fd88013a010 con 0x7fd880143be0 2024-03-29T11:09:05.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.359+0000 7fd8677fe640 10 monclient(hunting): _negotiate 2024-03-29T11:09:05.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.359+0000 7fd8677fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:05.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.359+0000 7fd8677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:05.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.359+0000 7fd8677fe640 10 monclient(hunting): my global_id is 4134 2024-03-29T11:09:05.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.359+0000 7fd8677fe640 1 -- 172.21.15.70:0/3427184638 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd88013a5d0 con 0x7fd880143be0 2024-03-29T11:09:05.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 1 -- 172.21.15.70:0/3427184638 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2247031279 0 0) 0x7fd86c004eb0 con 0x7fd8800b1780 2024-03-29T11:09:05.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 1 -- 172.21.15.70:0/3427184638 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd88013a010 con 0x7fd8800b1780 2024-03-29T11:09:05.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 1 -- 172.21.15.70:0/3427184638 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fd874001a10 con 0x7fd8800b1780 2024-03-29T11:09:05.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:05.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:09:05.353 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 10 monclient(hunting): dump: 2024-03-29T11:09:05.353 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:05.353 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:05.354 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.354 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.354 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:05.354 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:05.354 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:05.354 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:05.354 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:05.354 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:05.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:09:05.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 1 -- 172.21.15.70:0/3427184638 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 366672401 0 0) 0x7fd88013a2f0 con 0x7fd88013d7e0 2024-03-29T11:09:05.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 1 -- 172.21.15.70:0/3427184638 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd86c004eb0 con 0x7fd88013d7e0 2024-03-29T11:09:05.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 1 -- 172.21.15.70:0/3427184638 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fd870001b00 con 0x7fd88013d7e0 2024-03-29T11:09:05.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:05.354 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:09:05.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 10 monclient(hunting): dump: 2024-03-29T11:09:05.355 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:05.355 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:05.355 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.355 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.355 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:05.355 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:05.355 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:05.355 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:05.355 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:05.355 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:05.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 1 -- 172.21.15.70:0/3427184638 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1244610603 0 0) 0x7fd88013a5d0 con 0x7fd880143be0 2024-03-29T11:09:05.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 1 -- 172.21.15.70:0/3427184638 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd88013a2f0 con 0x7fd880143be0 2024-03-29T11:09:05.355 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 1 -- 172.21.15.70:0/3427184638 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fd878001b50 con 0x7fd880143be0 2024-03-29T11:09:05.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:05.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:09:05.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 10 monclient(hunting): dump: 2024-03-29T11:09:05.356 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:05.356 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:05.356 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.356 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.356 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:05.356 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:05.356 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:05.356 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:05.356 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:05.356 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:05.356 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 1 -- 172.21.15.70:0/3427184638 <== mon.1 v1:172.21.15.70:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3578445665 0 0) 0x7fd88013a010 con 0x7fd8800b1780 2024-03-29T11:09:05.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 1 -- 172.21.15.70:0/3427184638 >> v1:172.21.15.18:6790/0 conn(0x7fd88013d7e0 legacy=0x7fd880134990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:05.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 1 -- 172.21.15.70:0/3427184638 >> v1:172.21.15.18:6789/0 conn(0x7fd880143be0 legacy=0x7fd8801e2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:05.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 10 monclient: _finish_hunting 0 2024-03-29T11:09:05.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 1 monclient: found mon.b 2024-03-29T11:09:05.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:05.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:05.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 1 -- 172.21.15.70:0/3427184638 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8801d73e0 con 0x7fd8800b1780 2024-03-29T11:09:05.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 10 monclient: _finish_auth 0 2024-03-29T11:09:05.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 10 monclient: _check_auth_tickets 2024-03-29T11:09:05.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:05.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8850fb640 5 monclient: authenticate success, global_id 4132 2024-03-29T11:09:05.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8850fb640 10 monclient: _renew_subs 2024-03-29T11:09:05.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8850fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:05.357 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8850fb640 1 -- 172.21.15.70:0/3427184638 --> v1:172.21.15.70:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8801db570 con 0x7fd8800b1780 2024-03-29T11:09:05.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8850fb640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:09:05.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8850fb640 10 monclient: _renew_subs 2024-03-29T11:09:05.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8850fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:05.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8850fb640 1 -- 172.21.15.70:0/3427184638 --> v1:172.21.15.70:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd88006ca90 con 0x7fd8800b1780 2024-03-29T11:09:05.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8850fb640 1 librados: init done 2024-03-29T11:09:05.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 1 -- 172.21.15.70:0/3427184638 <== mon.1 v1:172.21.15.70:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd874001cf0 con 0x7fd8800b1780 2024-03-29T11:09:05.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:05.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 1 -- 172.21.15.70:0/3427184638 <== mon.1 v1:172.21.15.70:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fd874003a60 con 0x7fd8800b1780 2024-03-29T11:09:05.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:05.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:09:05.358 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 10 monclient: dump: 2024-03-29T11:09:05.358 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:05.358 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:05.358 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.359 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.359 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:05.359 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:05.359 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:05.359 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:05.359 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:05.359 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:05.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 1 -- 172.21.15.70:0/3427184638 <== mon.1 v1:172.21.15.70:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fd874004010 con 0x7fd8800b1780 2024-03-29T11:09:05.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd87d59a640 4 set_mon_vals no callback set 2024-03-29T11:09:05.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 1 -- 172.21.15.70:0/3427184638 <== mon.1 v1:172.21.15.70:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 4110881695 0 0) 0x7fd874005c50 con 0x7fd8800b1780 2024-03-29T11:09:05.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 10 client.4132.objecter ms_dispatch 0x7fd880000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T11:09:05.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 3 client.4132.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T11:09:05.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 3 client.4132.objecter handle_osd_map decoding full epoch 5 2024-03-29T11:09:05.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.360+0000 7fd8677fe640 20 client.4132.objecter dump_active .. 0 homeless 2024-03-29T11:09:05.359 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.361+0000 7fd8850fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:05.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.361+0000 7fd8850fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:05.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.361+0000 7fd8850fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:05.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.361+0000 7fd8850fb640 1 -- 172.21.15.70:0/3427184638 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd860005080 con 0x7fd8800b1780 2024-03-29T11:09:05.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.363+0000 7fd8677fe640 1 -- 172.21.15.70:0/3427184638 <== mon.1 v1:172.21.15.70:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fd8740042c0 con 0x7fd8800b1780 2024-03-29T11:09:05.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.363+0000 7fd8677fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:05.360 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.363+0000 7fd8677fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:09:05.360 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-29T11:09:05.373+0000 7fad3de2e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:09:05.396 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-29T11:09:05.417+0000 7fad3de2e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:09:05.403 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-03-29T11:09:05.424+0000 7f6963a2c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:09:05.432 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.454+0000 7fd8850fb640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c74142fb-d4f2-4e57-a784-70d935727b13", "id": 3}] 2024-03-29T11:09:05.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.454+0000 7fd8850fb640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c74142fb-d4f2-4e57-a784-70d935727b13", "id": 3}] 2024-03-29T11:09:05.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.454+0000 7fd8850fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:05.433 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.454+0000 7fd8850fb640 1 -- 172.21.15.70:0/3427184638 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "osd new", "uuid": "c74142fb-d4f2-4e57-a784-70d935727b13", "id": 3} v 0) v1 -- 0x7fd8600058b0 con 0x7fd8800b1780 2024-03-29T11:09:05.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.457+0000 7fd8677fe640 1 -- 172.21.15.70:0/3427184638 <== mon.1 v1:172.21.15.70:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c74142fb-d4f2-4e57-a784-70d935727b13", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 2422121882 0 525046306) 0x7fd8740057c0 con 0x7fd8800b1780 2024-03-29T11:09:05.435 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.457+0000 7fd8677fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c74142fb-d4f2-4e57-a784-70d935727b13", "id": 3}] 2024-03-29T11:09:05.436 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.457+0000 7fd8677fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:09:05.436 INFO:teuthology.orchestra.run.smithi070.stdout:3 2024-03-29T11:09:05.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.458+0000 7fd8850fb640 10 librados: watch_flush enter 2024-03-29T11:09:05.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.458+0000 7fd8850fb640 10 librados: watch_flush exit 2024-03-29T11:09:05.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.458+0000 7fd8850fb640 20 client.4132.objecter shutdown clearing up homeless session... 2024-03-29T11:09:05.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.458+0000 7fd8850fb640 10 client.4132.objecter successfully canceled tick 2024-03-29T11:09:05.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.458+0000 7fd8850fb640 10 monclient: shutdown 2024-03-29T11:09:05.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.458+0000 7fd8850fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:05.437 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.458+0000 7fd8850fb640 1 -- 172.21.15.70:0/3427184638 >> v1:172.21.15.70:6789/0 conn(0x7fd8800b1780 legacy=0x7fd8801dae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:05.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.458+0000 7fd8850fb640 1 -- 172.21.15.70:0/3427184638 shutdown_connections 2024-03-29T11:09:05.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.458+0000 7fd8850fb640 1 -- 172.21.15.70:0/3427184638 >> 172.21.15.70:0/3427184638 conn(0x7fd8800b58f0 msgr2=0x7fd880069bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:05.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.458+0000 7fd8850fb640 1 -- 172.21.15.70:0/3427184638 shutdown_connections 2024-03-29T11:09:05.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.458+0000 7fd8850fb640 1 -- 172.21.15.70:0/3427184638 wait complete. 2024-03-29T11:09:05.438 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.458+0000 7fd8850fb640 1 librados: shutdown 2024-03-29T11:09:05.447 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --cluster ceph osd new 1d9bfdeb-5be9-4106-86dc-00841b3419f2 4 2024-03-29T11:09:05.447 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-03-29T11:09:05.469+0000 7f6963a2c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:09:05.517 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-29T11:09:05.538+0000 7fad3de2e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:09:05.545 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.567+0000 7fa3ef69a640 10 monclient: get_monmap_and_config 2024-03-29T11:09:05.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.567+0000 7fa3ef69a640 10 monclient: build_initial_monmap 2024-03-29T11:09:05.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.567+0000 7fa3ef69a640 10 monclient: monmap: 2024-03-29T11:09:05.546 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-03-29T11:09:05.546 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:05.546 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:09:05.568178+0000 2024-03-29T11:09:05.546 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:09:05.568178+0000 2024-03-29T11:09:05.546 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:05.546 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:05.546 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:09:05.546 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:09:05.546 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:09:05.546 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:05.546 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.567+0000 7fa3ef69a640 1 Processor -- start 2024-03-29T11:09:05.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.567+0000 7fa3ef69a640 1 -- start start 2024-03-29T11:09:05.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.567+0000 7fa3ef69a640 10 monclient: init 2024-03-29T11:09:05.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.568+0000 7fa3ef69a640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:05.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.568+0000 7fa3ef69a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:09:05.555 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.568+0000 7fa3ef69a640 10 monclient(hunting): picked mon.noname-b con 0x7fa3e809ac40 addr v1:172.21.15.18:6790/0 2024-03-29T11:09:05.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.568+0000 7fa3ef69a640 10 monclient(hunting): picked mon.noname-a con 0x7fa3e809c870 addr v1:172.21.15.18:6789/0 2024-03-29T11:09:05.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.568+0000 7fa3ef69a640 10 monclient(hunting): picked mon.noname-c con 0x7fa3e80b4360 addr v1:172.21.15.70:6789/0 2024-03-29T11:09:05.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.568+0000 7fa3ef69a640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e809dec0 con 0x7fa3e809c870 2024-03-29T11:09:05.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.568+0000 7fa3ef69a640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e809da30 con 0x7fa3e80b4360 2024-03-29T11:09:05.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.568+0000 7fa3ef69a640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e80b39c0 con 0x7fa3e809ac40 2024-03-29T11:09:05.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.568+0000 7fa3ef69a640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:05.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.568+0000 7fa3ef69a640 10 monclient(hunting): authenticate will time out at 761.796787s 2024-03-29T11:09:05.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.568+0000 7fa3edc37640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fa3e80b4360 0x7fa3e814d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:48256/0 (socket says 172.21.15.70:48256) 2024-03-29T11:09:05.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.568+0000 7fa3edc37640 1 -- 172.21.15.70:0/1135234321 learned_addr learned my addr 172.21.15.70:0/1135234321 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-03-29T11:09:05.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.568+0000 7fa3dffff640 1 -- 172.21.15.70:0/1135234321 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2566650576 0 0) 0x7fa3e809da30 con 0x7fa3e80b4360 2024-03-29T11:09:05.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.568+0000 7fa3dffff640 10 monclient(hunting): _negotiate 2024-03-29T11:09:05.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.568+0000 7fa3dffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:05.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.568+0000 7fa3dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:05.556 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.568+0000 7fa3dffff640 10 monclient(hunting): my global_id is 4135 2024-03-29T11:09:05.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.568+0000 7fa3dffff640 1 -- 172.21.15.70:0/1135234321 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3d4003520 con 0x7fa3e80b4360 2024-03-29T11:09:05.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.569+0000 7fa3dffff640 1 -- 172.21.15.70:0/1135234321 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 379858220 0 0) 0x7fa3d8001040 con 0x7fa3e809c870 2024-03-29T11:09:05.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.569+0000 7fa3dffff640 10 monclient(hunting): _negotiate 2024-03-29T11:09:05.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.569+0000 7fa3dffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:05.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.569+0000 7fa3dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:05.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.569+0000 7fa3dffff640 10 monclient(hunting): my global_id is 4137 2024-03-29T11:09:05.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.569+0000 7fa3dffff640 1 -- 172.21.15.70:0/1135234321 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3e809da30 con 0x7fa3e809c870 2024-03-29T11:09:05.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.569+0000 7fa3dffff640 1 -- 172.21.15.70:0/1135234321 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4119575264 0 0) 0x7fa3e80b39c0 con 0x7fa3e809ac40 2024-03-29T11:09:05.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.569+0000 7fa3dffff640 10 monclient(hunting): _negotiate 2024-03-29T11:09:05.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.569+0000 7fa3dffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:05.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.569+0000 7fa3dffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:05.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.569+0000 7fa3dffff640 10 monclient(hunting): my global_id is 4139 2024-03-29T11:09:05.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.569+0000 7fa3dffff640 1 -- 172.21.15.70:0/1135234321 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3d8001040 con 0x7fa3e809ac40 2024-03-29T11:09:05.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.569+0000 7fa3dffff640 1 -- 172.21.15.70:0/1135234321 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2246345567 0 0) 0x7fa3d4003520 con 0x7fa3e80b4360 2024-03-29T11:09:05.557 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.569+0000 7fa3dffff640 1 -- 172.21.15.70:0/1135234321 >> v1:172.21.15.18:6790/0 conn(0x7fa3e809ac40 legacy=0x7fa3e80b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:05.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.569+0000 7fa3dffff640 1 -- 172.21.15.70:0/1135234321 >> v1:172.21.15.18:6789/0 conn(0x7fa3e809c870 legacy=0x7fa3e813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:05.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.569+0000 7fa3dffff640 10 monclient: _finish_hunting 0 2024-03-29T11:09:05.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.569+0000 7fa3dffff640 1 monclient: found mon.noname-c 2024-03-29T11:09:05.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.569+0000 7fa3dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:05.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.569+0000 7fa3dffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.70:6789/0 2024-03-29T11:09:05.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.569+0000 7fa3dffff640 1 -- 172.21.15.70:0/1135234321 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e80b5160 con 0x7fa3e80b4360 2024-03-29T11:09:05.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.569+0000 7fa3dffff640 10 monclient: _finish_auth 0 2024-03-29T11:09:05.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.569+0000 7fa3dffff640 10 monclient: _check_auth_tickets 2024-03-29T11:09:05.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.569+0000 7fa3dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:05.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.569+0000 7fa3ef69a640 5 monclient: authenticate success, global_id 4135 2024-03-29T11:09:05.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.569+0000 7fa3dffff640 1 -- 172.21.15.70:0/1135234321 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fa3e4001690 con 0x7fa3e80b4360 2024-03-29T11:09:05.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.569+0000 7fa3dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:05.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.569+0000 7fa3dffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:09:05.558 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.569+0000 7fa3dffff640 10 monclient: dump: 2024-03-29T11:09:05.559 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:05.559 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:05.559 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.559 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.559 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:05.559 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:05.559 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:05.559 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:05.559 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:05.559 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:05.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.569+0000 7fa3ef69a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:05.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.569+0000 7fa3dffff640 1 -- 172.21.15.70:0/1135234321 <== mon.1 v1:172.21.15.70:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa3e4001870 con 0x7fa3e80b4360 2024-03-29T11:09:05.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.569+0000 7fa3dffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:05.559 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.569+0000 7fa3dffff640 1 -- 172.21.15.70:0/1135234321 <== mon.1 v1:172.21.15.70:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fa3e4001c80 con 0x7fa3e80b4360 2024-03-29T11:09:05.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.569+0000 7fa3dffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:05.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.569+0000 7fa3dffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:09:05.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.569+0000 7fa3dffff640 10 monclient: dump: 2024-03-29T11:09:05.560 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:05.560 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:05.560 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.560 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.560 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:05.560 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:05.560 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:05.560 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:05.560 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:05.560 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:05.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.569+0000 7fa3ef69a640 10 monclient: get_monmap_and_config success 2024-03-29T11:09:05.560 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.569+0000 7fa3ef69a640 4 set_mon_vals no callback set 2024-03-29T11:09:05.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.570+0000 7fa3ef69a640 10 monclient: shutdown 2024-03-29T11:09:05.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.570+0000 7fa3ef69a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:05.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.570+0000 7fa3ef69a640 1 -- 172.21.15.70:0/1135234321 >> v1:172.21.15.70:6789/0 conn(0x7fa3e80b4360 legacy=0x7fa3e814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:05.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.570+0000 7fa3ef69a640 1 -- 172.21.15.70:0/1135234321 shutdown_connections 2024-03-29T11:09:05.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.570+0000 7fa3ef69a640 1 -- 172.21.15.70:0/1135234321 >> 172.21.15.70:0/1135234321 conn(0x7fa3e8066b00 msgr2=0x7fa3e8066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:05.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.570+0000 7fa3ef69a640 1 -- 172.21.15.70:0/1135234321 shutdown_connections 2024-03-29T11:09:05.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.570+0000 7fa3ef69a640 1 -- 172.21.15.70:0/1135234321 wait complete. 2024-03-29T11:09:05.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.571+0000 7fa3ef69a640 10 monclient: build_initial_monmap 2024-03-29T11:09:05.561 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.571+0000 7fa3ef69a640 10 monclient: monmap: 2024-03-29T11:09:05.561 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-03-29T11:09:05.561 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:09:05.561 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-03-29T11:09:05.561 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-03-29T11:09:05.561 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:05.561 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:05.562 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:09:05.562 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:09:05.562 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:09:05.562 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:05.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.571+0000 7fa3ef69a640 1 librados: starting msgr at 2024-03-29T11:09:05.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.571+0000 7fa3ef69a640 1 librados: starting objecter 2024-03-29T11:09:05.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.571+0000 7fa3ef69a640 1 Processor -- start 2024-03-29T11:09:05.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.571+0000 7fa3ef69a640 1 -- start start 2024-03-29T11:09:05.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.571+0000 7fa3ef69a640 1 librados: setting wanted keys 2024-03-29T11:09:05.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.571+0000 7fa3ef69a640 1 librados: calling monclient init 2024-03-29T11:09:05.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.571+0000 7fa3ef69a640 10 monclient: init 2024-03-29T11:09:05.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.572+0000 7fa3ef69a640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:05.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.572+0000 7fa3ef69a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:09:05.562 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.572+0000 7fa3ef69a640 10 monclient(hunting): picked mon.noname-c con 0x7fa3e809ac40 addr v1:172.21.15.18:6790/0 2024-03-29T11:09:05.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.572+0000 7fa3ef69a640 10 monclient(hunting): picked mon.noname-b con 0x7fa3e813ffa0 addr v1:172.21.15.70:6789/0 2024-03-29T11:09:05.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.572+0000 7fa3ef69a640 10 monclient(hunting): picked mon.noname-a con 0x7fa3e8145a90 addr v1:172.21.15.18:6789/0 2024-03-29T11:09:05.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.572+0000 7fa3ef69a640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e8140530 con 0x7fa3e8145a90 2024-03-29T11:09:05.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.572+0000 7fa3ef69a640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e8137850 con 0x7fa3e813ffa0 2024-03-29T11:09:05.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.572+0000 7fa3ef69a640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e8141290 con 0x7fa3e809ac40 2024-03-29T11:09:05.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.572+0000 7fa3ef69a640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:05.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.572+0000 7fa3ef69a640 10 monclient(hunting): authenticate will time out at 761.800823s 2024-03-29T11:09:05.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.572+0000 7fa3ecc35640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fa3e813ffa0 0x7fa3e8136a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:48258/0 (socket says 172.21.15.70:48258) 2024-03-29T11:09:05.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.572+0000 7fa3ecc35640 1 -- 172.21.15.70:0/708694424 learned_addr learned my addr 172.21.15.70:0/708694424 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-03-29T11:09:05.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.572+0000 7fa3ddffb640 10 client.?.objecter ms_handle_connect 0x7fa3e813ffa0 2024-03-29T11:09:05.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.572+0000 7fa3ddffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:05.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.572+0000 7fa3ddffb640 10 client.?.objecter ms_handle_connect 0x7fa3e809ac40 2024-03-29T11:09:05.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.572+0000 7fa3ddffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:05.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.572+0000 7fa3ddffb640 10 client.?.objecter ms_handle_connect 0x7fa3e8145a90 2024-03-29T11:09:05.563 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.572+0000 7fa3ddffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:05.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.572+0000 7fa3ddffb640 1 -- 172.21.15.70:0/708694424 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3585594425 0 0) 0x7fa3e8137850 con 0x7fa3e813ffa0 2024-03-29T11:09:05.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.572+0000 7fa3ddffb640 10 monclient(hunting): _negotiate 2024-03-29T11:09:05.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.572+0000 7fa3ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:05.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.572+0000 7fa3ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:05.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.572+0000 7fa3ddffb640 10 monclient(hunting): my global_id is 4138 2024-03-29T11:09:05.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.572+0000 7fa3ddffb640 1 -- 172.21.15.70:0/708694424 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3cc004eb0 con 0x7fa3e813ffa0 2024-03-29T11:09:05.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.572+0000 7fa3ddffb640 1 -- 172.21.15.70:0/708694424 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1994327041 0 0) 0x7fa3e8141290 con 0x7fa3e809ac40 2024-03-29T11:09:05.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.572+0000 7fa3ddffb640 10 monclient(hunting): _negotiate 2024-03-29T11:09:05.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.572+0000 7fa3ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:05.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.572+0000 7fa3ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:05.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.572+0000 7fa3ddffb640 10 monclient(hunting): my global_id is 4142 2024-03-29T11:09:05.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 1 -- 172.21.15.70:0/708694424 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3e8137850 con 0x7fa3e809ac40 2024-03-29T11:09:05.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 1 -- 172.21.15.70:0/708694424 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3474374355 0 0) 0x7fa3e8140530 con 0x7fa3e8145a90 2024-03-29T11:09:05.564 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 10 monclient(hunting): _negotiate 2024-03-29T11:09:05.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:05.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:05.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 10 monclient(hunting): my global_id is 4140 2024-03-29T11:09:05.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 1 -- 172.21.15.70:0/708694424 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3e8141290 con 0x7fa3e8145a90 2024-03-29T11:09:05.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 1 -- 172.21.15.70:0/708694424 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1377598758 0 0) 0x7fa3cc004eb0 con 0x7fa3e813ffa0 2024-03-29T11:09:05.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 1 -- 172.21.15.70:0/708694424 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3e8140530 con 0x7fa3e813ffa0 2024-03-29T11:09:05.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 1 -- 172.21.15.70:0/708694424 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fa3d8001950 con 0x7fa3e813ffa0 2024-03-29T11:09:05.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:05.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:09:05.565 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 10 monclient(hunting): dump: 2024-03-29T11:09:05.565 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:05.565 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:05.565 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.565 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.566 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:05.566 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:05.566 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:05.566 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:05.566 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:05.566 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:05.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:09:05.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 1 -- 172.21.15.70:0/708694424 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3827174365 0 0) 0x7fa3e8141290 con 0x7fa3e8145a90 2024-03-29T11:09:05.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 1 -- 172.21.15.70:0/708694424 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3cc004eb0 con 0x7fa3e8145a90 2024-03-29T11:09:05.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 1 -- 172.21.15.70:0/708694424 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1673094705 0 0) 0x7fa3e8137850 con 0x7fa3e809ac40 2024-03-29T11:09:05.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 1 -- 172.21.15.70:0/708694424 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa3e8141290 con 0x7fa3e809ac40 2024-03-29T11:09:05.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 1 -- 172.21.15.70:0/708694424 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fa3e4001b80 con 0x7fa3e8145a90 2024-03-29T11:09:05.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:05.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:09:05.566 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 10 monclient(hunting): dump: 2024-03-29T11:09:05.567 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:05.567 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:05.567 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.567 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.567 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:05.567 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:05.567 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:05.567 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:05.567 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:05.567 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:05.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 1 -- 172.21.15.70:0/708694424 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fa3e0001bb0 con 0x7fa3e809ac40 2024-03-29T11:09:05.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:05.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:09:05.567 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 10 monclient(hunting): dump: 2024-03-29T11:09:05.568 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:05.568 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:05.568 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.568 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.568 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:05.568 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:05.568 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:05.568 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:05.568 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:05.568 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:05.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 1 -- 172.21.15.70:0/708694424 <== mon.1 v1:172.21.15.70:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3425113793 0 0) 0x7fa3e8140530 con 0x7fa3e813ffa0 2024-03-29T11:09:05.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 1 -- 172.21.15.70:0/708694424 >> v1:172.21.15.18:6790/0 conn(0x7fa3e809ac40 legacy=0x7fa3e80ae130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:05.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 1 -- 172.21.15.70:0/708694424 >> v1:172.21.15.18:6789/0 conn(0x7fa3e8145a90 legacy=0x7fa3e8137140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:05.568 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 10 monclient: _finish_hunting 0 2024-03-29T11:09:05.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 1 monclient: found mon.b 2024-03-29T11:09:05.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:05.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:05.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 1 -- 172.21.15.70:0/708694424 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e81daeb0 con 0x7fa3e813ffa0 2024-03-29T11:09:05.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 10 monclient: _finish_auth 0 2024-03-29T11:09:05.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 10 monclient: _check_auth_tickets 2024-03-29T11:09:05.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:05.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ef69a640 5 monclient: authenticate success, global_id 4138 2024-03-29T11:09:05.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ef69a640 10 monclient: _renew_subs 2024-03-29T11:09:05.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ef69a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:05.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ef69a640 1 -- 172.21.15.70:0/708694424 --> v1:172.21.15.70:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3e81db040 con 0x7fa3e813ffa0 2024-03-29T11:09:05.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 1 -- 172.21.15.70:0/708694424 <== mon.1 v1:172.21.15.70:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa3d8003610 con 0x7fa3e813ffa0 2024-03-29T11:09:05.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:05.569 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 1 -- 172.21.15.70:0/708694424 <== mon.1 v1:172.21.15.70:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fa3d80039e0 con 0x7fa3e813ffa0 2024-03-29T11:09:05.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:05.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:09:05.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 10 monclient: dump: 2024-03-29T11:09:05.570 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:05.570 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:05.570 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.570 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.570 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:05.570 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:05.570 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:05.570 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:05.570 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:05.570 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:05.570 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 1 -- 172.21.15.70:0/708694424 <== mon.1 v1:172.21.15.70:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fa3d8003f50 con 0x7fa3e813ffa0 2024-03-29T11:09:05.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ef69a640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:09:05.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ef69a640 10 monclient: _renew_subs 2024-03-29T11:09:05.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3df7fe640 4 set_mon_vals no callback set 2024-03-29T11:09:05.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ef69a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:05.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ef69a640 1 -- 172.21.15.70:0/708694424 --> v1:172.21.15.70:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3e81db8d0 con 0x7fa3e813ffa0 2024-03-29T11:09:05.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ef69a640 1 librados: init done 2024-03-29T11:09:05.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 1 -- 172.21.15.70:0/708694424 <== mon.1 v1:172.21.15.70:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 352974202 0 0) 0x7fa3d80043e0 con 0x7fa3e813ffa0 2024-03-29T11:09:05.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 10 client.4138.objecter ms_dispatch 0x7fa3e8000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T11:09:05.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 3 client.4138.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T11:09:05.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 3 client.4138.objecter handle_osd_map decoding full epoch 6 2024-03-29T11:09:05.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.573+0000 7fa3ddffb640 20 client.4138.objecter dump_active .. 0 homeless 2024-03-29T11:09:05.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.574+0000 7fa3ef69a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:05.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.574+0000 7fa3ef69a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:05.571 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.574+0000 7fa3ef69a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:05.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.574+0000 7fa3ef69a640 1 -- 172.21.15.70:0/708694424 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3e81db390 con 0x7fa3e813ffa0 2024-03-29T11:09:05.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.576+0000 7fa3ddffb640 1 -- 172.21.15.70:0/708694424 <== mon.1 v1:172.21.15.70:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fa3d80057c0 con 0x7fa3e813ffa0 2024-03-29T11:09:05.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.576+0000 7fa3ddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:05.572 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.576+0000 7fa3ddffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:09:05.572 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-03-29T11:09:05.590+0000 7f6963a2c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:09:05.590 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-29T11:09:05.611+0000 7fad3de2e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:09:05.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.646+0000 7fa3ef69a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "1d9bfdeb-5be9-4106-86dc-00841b3419f2", "id": 4}] 2024-03-29T11:09:05.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.646+0000 7fa3ef69a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "1d9bfdeb-5be9-4106-86dc-00841b3419f2", "id": 4}] 2024-03-29T11:09:05.624 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.646+0000 7fa3ef69a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:05.625 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.646+0000 7fa3ef69a640 1 -- 172.21.15.70:0/708694424 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "osd new", "uuid": "1d9bfdeb-5be9-4106-86dc-00841b3419f2", "id": 4} v 0) v1 -- 0x7fa3e81dbb80 con 0x7fa3e813ffa0 2024-03-29T11:09:05.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.648+0000 7fa3ddffb640 1 -- 172.21.15.70:0/708694424 <== mon.1 v1:172.21.15.70:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1d9bfdeb-5be9-4106-86dc-00841b3419f2", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 3197037989 0 1697012583) 0x7fa3d80059e0 con 0x7fa3e813ffa0 2024-03-29T11:09:05.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.648+0000 7fa3ddffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "1d9bfdeb-5be9-4106-86dc-00841b3419f2", "id": 4}] 2024-03-29T11:09:05.627 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.648+0000 7fa3ddffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:09:05.627 INFO:teuthology.orchestra.run.smithi070.stdout:4 2024-03-29T11:09:05.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.649+0000 7fa3ef69a640 10 librados: watch_flush enter 2024-03-29T11:09:05.628 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.649+0000 7fa3ef69a640 10 librados: watch_flush exit 2024-03-29T11:09:05.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.649+0000 7fa3ef69a640 20 client.4138.objecter shutdown clearing up homeless session... 2024-03-29T11:09:05.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.649+0000 7fa3ef69a640 10 client.4138.objecter successfully canceled tick 2024-03-29T11:09:05.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.649+0000 7fa3ef69a640 10 monclient: shutdown 2024-03-29T11:09:05.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.649+0000 7fa3ef69a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:05.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.649+0000 7fa3ef69a640 1 -- 172.21.15.70:0/708694424 >> v1:172.21.15.70:6789/0 conn(0x7fa3e813ffa0 legacy=0x7fa3e8136a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:05.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.649+0000 7fa3ef69a640 1 -- 172.21.15.70:0/708694424 shutdown_connections 2024-03-29T11:09:05.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.649+0000 7fa3ef69a640 1 -- 172.21.15.70:0/708694424 >> 172.21.15.70:0/708694424 conn(0x7fa3e8066b00 msgr2=0x7fa3e813baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:05.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.649+0000 7fa3ef69a640 1 -- 172.21.15.70:0/708694424 shutdown_connections 2024-03-29T11:09:05.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.649+0000 7fa3ef69a640 1 -- 172.21.15.70:0/708694424 wait complete. 2024-03-29T11:09:05.629 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.650+0000 7fa3ef69a640 1 librados: shutdown 2024-03-29T11:09:05.638 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --cluster ceph osd new 4d29e43e-9f99-450d-9d20-6b8dbf6eaa34 5 2024-03-29T11:09:05.639 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-29T11:09:05.660+0000 7fad3de2e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:09:05.639 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-03-29T11:09:05.661+0000 7f6963a2c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:09:05.688 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-03-29T11:09:05.709+0000 7f6963a2c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:09:05.693 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-29T11:09:05.715+0000 7fad3de2e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:09:05.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.759+0000 7f15a624a640 10 monclient: get_monmap_and_config 2024-03-29T11:09:05.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.759+0000 7f15a624a640 10 monclient: build_initial_monmap 2024-03-29T11:09:05.737 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.759+0000 7f15a624a640 10 monclient: monmap: 2024-03-29T11:09:05.737 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-03-29T11:09:05.738 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:05.738 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:09:05.760137+0000 2024-03-29T11:09:05.738 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:09:05.760137+0000 2024-03-29T11:09:05.738 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:05.738 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:05.738 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:09:05.738 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:09:05.738 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:09:05.738 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:05.745 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.759+0000 7f15a624a640 1 Processor -- start 2024-03-29T11:09:05.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.759+0000 7f15a624a640 1 -- start start 2024-03-29T11:09:05.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.759+0000 7f15a624a640 10 monclient: init 2024-03-29T11:09:05.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.760+0000 7f15a624a640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:05.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.760+0000 7f15a624a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:09:05.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.760+0000 7f15a624a640 10 monclient(hunting): picked mon.noname-c con 0x7f15a0066b90 addr v1:172.21.15.70:6789/0 2024-03-29T11:09:05.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.760+0000 7f15a624a640 10 monclient(hunting): picked mon.noname-a con 0x7f15a013f340 addr v1:172.21.15.18:6789/0 2024-03-29T11:09:05.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.760+0000 7f15a624a640 10 monclient(hunting): picked mon.noname-b con 0x7f15a013f7a0 addr v1:172.21.15.18:6790/0 2024-03-29T11:09:05.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.760+0000 7f15a624a640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a013fc30 con 0x7f15a013f340 2024-03-29T11:09:05.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.760+0000 7f15a624a640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a00afd70 con 0x7f15a0066b90 2024-03-29T11:09:05.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.760+0000 7f15a624a640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a00aec00 con 0x7f15a013f7a0 2024-03-29T11:09:05.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.760+0000 7f15a624a640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:05.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.760+0000 7f15a624a640 10 monclient(hunting): authenticate will time out at 761.988946s 2024-03-29T11:09:05.746 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.760+0000 7f159f7fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f15a0066b90 0x7f15a00af580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:48264/0 (socket says 172.21.15.70:48264) 2024-03-29T11:09:05.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.760+0000 7f159effd640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f15a013f340 0x7f15a0142060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.70:35576/0 (socket says 172.21.15.70:35576) 2024-03-29T11:09:05.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.760+0000 7f159ffff640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f15a013f7a0 0x7f15a0142770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.70:37740/0 (socket says 172.21.15.70:37740) 2024-03-29T11:09:05.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.760+0000 7f159f7fe640 1 -- 172.21.15.70:0/1722081677 learned_addr learned my addr 172.21.15.70:0/1722081677 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-03-29T11:09:05.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f159e7fc640 1 -- 172.21.15.70:0/1722081677 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1478807052 0 0) 0x7f15a00afd70 con 0x7f15a0066b90 2024-03-29T11:09:05.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f159e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:09:05.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f159e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:05.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f159e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:05.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f159e7fc640 10 monclient(hunting): my global_id is 4141 2024-03-29T11:09:05.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f159e7fc640 1 -- 172.21.15.70:0/1722081677 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1588003520 con 0x7f15a0066b90 2024-03-29T11:09:05.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f159e7fc640 1 -- 172.21.15.70:0/1722081677 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3903634625 0 0) 0x7f15a013fc30 con 0x7f15a013f340 2024-03-29T11:09:05.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f159e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:09:05.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f159e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:05.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f159e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:05.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f159e7fc640 10 monclient(hunting): my global_id is 4143 2024-03-29T11:09:05.747 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f159e7fc640 1 -- 172.21.15.70:0/1722081677 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15a00afd70 con 0x7f15a013f340 2024-03-29T11:09:05.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f159e7fc640 1 -- 172.21.15.70:0/1722081677 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2615242513 0 0) 0x7f15a00aec00 con 0x7f15a013f7a0 2024-03-29T11:09:05.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f159e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:09:05.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f159e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:05.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f159e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:05.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f159e7fc640 10 monclient(hunting): my global_id is 4145 2024-03-29T11:09:05.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f159e7fc640 1 -- 172.21.15.70:0/1722081677 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15a013fc30 con 0x7f15a013f7a0 2024-03-29T11:09:05.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f159e7fc640 1 -- 172.21.15.70:0/1722081677 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 861425540 0 0) 0x7f1588003520 con 0x7f15a0066b90 2024-03-29T11:09:05.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f159e7fc640 1 -- 172.21.15.70:0/1722081677 >> v1:172.21.15.18:6790/0 conn(0x7f15a013f7a0 legacy=0x7f15a0142770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:05.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f159e7fc640 1 -- 172.21.15.70:0/1722081677 >> v1:172.21.15.18:6789/0 conn(0x7f15a013f340 legacy=0x7f15a0142060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:05.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f159e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:09:05.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f159e7fc640 1 monclient: found mon.noname-c 2024-03-29T11:09:05.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f159e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:05.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f159e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.70:6789/0 2024-03-29T11:09:05.748 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f159e7fc640 1 -- 172.21.15.70:0/1722081677 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a00aed90 con 0x7f15a0066b90 2024-03-29T11:09:05.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f159e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:09:05.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f159e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:09:05.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f159e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:05.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f159e7fc640 1 -- 172.21.15.70:0/1722081677 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f1590001690 con 0x7f15a0066b90 2024-03-29T11:09:05.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f15a624a640 5 monclient: authenticate success, global_id 4141 2024-03-29T11:09:05.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f15a624a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:05.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f159e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:05.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f159e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:09:05.749 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f159e7fc640 10 monclient: dump: 2024-03-29T11:09:05.749 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:05.749 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:05.749 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.749 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.749 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:05.750 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:05.750 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:05.750 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:05.750 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:05.750 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:05.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f15a624a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:05.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f159e7fc640 1 -- 172.21.15.70:0/1722081677 <== mon.1 v1:172.21.15.70:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1590001870 con 0x7f15a0066b90 2024-03-29T11:09:05.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f159e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:05.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f159e7fc640 1 -- 172.21.15.70:0/1722081677 <== mon.1 v1:172.21.15.70:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f1590001cc0 con 0x7f15a0066b90 2024-03-29T11:09:05.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f15a624a640 10 monclient: get_monmap_and_config success 2024-03-29T11:09:05.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f15a624a640 4 set_mon_vals no callback set 2024-03-29T11:09:05.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f159e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:05.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f159e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:09:05.750 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.761+0000 7f159e7fc640 10 monclient: dump: 2024-03-29T11:09:05.751 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:05.751 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:05.751 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.751 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.751 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:05.751 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:05.751 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:05.751 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:05.751 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:05.751 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:05.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.762+0000 7f15a624a640 10 monclient: shutdown 2024-03-29T11:09:05.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.762+0000 7f15a624a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:05.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.762+0000 7f15a624a640 1 -- 172.21.15.70:0/1722081677 >> v1:172.21.15.70:6789/0 conn(0x7f15a0066b90 legacy=0x7f15a00af580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:05.751 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.762+0000 7f15a624a640 1 -- 172.21.15.70:0/1722081677 shutdown_connections 2024-03-29T11:09:05.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.762+0000 7f15a624a640 1 -- 172.21.15.70:0/1722081677 >> 172.21.15.70:0/1722081677 conn(0x7f15a00b4970 msgr2=0x7f15a00b4d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:05.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.762+0000 7f15a624a640 1 -- 172.21.15.70:0/1722081677 shutdown_connections 2024-03-29T11:09:05.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.762+0000 7f15a624a640 1 -- 172.21.15.70:0/1722081677 wait complete. 2024-03-29T11:09:05.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.762+0000 7f15a624a640 10 monclient: build_initial_monmap 2024-03-29T11:09:05.752 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.762+0000 7f15a624a640 10 monclient: monmap: 2024-03-29T11:09:05.752 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-03-29T11:09:05.752 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:09:05.752 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-03-29T11:09:05.752 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-03-29T11:09:05.752 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:05.752 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:05.752 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:09:05.752 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:09:05.752 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:09:05.752 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:05.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.763+0000 7f15a624a640 1 librados: starting msgr at 2024-03-29T11:09:05.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.763+0000 7f15a624a640 1 librados: starting objecter 2024-03-29T11:09:05.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.763+0000 7f15a624a640 1 Processor -- start 2024-03-29T11:09:05.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.763+0000 7f15a624a640 1 -- start start 2024-03-29T11:09:05.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.763+0000 7f15a624a640 1 librados: setting wanted keys 2024-03-29T11:09:05.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.763+0000 7f15a624a640 1 librados: calling monclient init 2024-03-29T11:09:05.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.763+0000 7f15a624a640 10 monclient: init 2024-03-29T11:09:05.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.763+0000 7f15a624a640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:05.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.763+0000 7f15a624a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:09:05.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.763+0000 7f15a624a640 10 monclient(hunting): picked mon.noname-c con 0x7f15a0066b90 addr v1:172.21.15.18:6790/0 2024-03-29T11:09:05.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.763+0000 7f15a624a640 10 monclient(hunting): picked mon.noname-b con 0x7f15a013f340 addr v1:172.21.15.70:6789/0 2024-03-29T11:09:05.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.763+0000 7f15a624a640 10 monclient(hunting): picked mon.noname-a con 0x7f15a013f7a0 addr v1:172.21.15.18:6789/0 2024-03-29T11:09:05.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.763+0000 7f15a624a640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a0070be0 con 0x7f15a013f7a0 2024-03-29T11:09:05.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.763+0000 7f15a624a640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a0140a50 con 0x7f15a013f340 2024-03-29T11:09:05.753 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.763+0000 7f15a624a640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a0070390 con 0x7f15a0066b90 2024-03-29T11:09:05.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.763+0000 7f15a624a640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:05.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.763+0000 7f15a624a640 10 monclient(hunting): authenticate will time out at 761.992089s 2024-03-29T11:09:05.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.763+0000 7f159effd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f15a013f340 0x7f15a0071090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:48270/0 (socket says 172.21.15.70:48270) 2024-03-29T11:09:05.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.763+0000 7f159effd640 1 -- 172.21.15.70:0/1326546890 learned_addr learned my addr 172.21.15.70:0/1326546890 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-03-29T11:09:05.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.763+0000 7f1587fff640 10 client.?.objecter ms_handle_connect 0x7f15a013f340 2024-03-29T11:09:05.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:05.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 10 client.?.objecter ms_handle_connect 0x7f15a0066b90 2024-03-29T11:09:05.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:05.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 10 client.?.objecter ms_handle_connect 0x7f15a013f7a0 2024-03-29T11:09:05.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:05.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 1 -- 172.21.15.70:0/1326546890 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3063407431 0 0) 0x7f15a0140a50 con 0x7f15a013f340 2024-03-29T11:09:05.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 10 monclient(hunting): _negotiate 2024-03-29T11:09:05.754 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:05.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:05.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 10 monclient(hunting): my global_id is 4144 2024-03-29T11:09:05.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 1 -- 172.21.15.70:0/1326546890 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f158c004eb0 con 0x7f15a013f340 2024-03-29T11:09:05.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 1 -- 172.21.15.70:0/1326546890 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 472472701 0 0) 0x7f15a0070390 con 0x7f15a0066b90 2024-03-29T11:09:05.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 10 monclient(hunting): _negotiate 2024-03-29T11:09:05.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:05.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:05.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 10 monclient(hunting): my global_id is 4148 2024-03-29T11:09:05.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 1 -- 172.21.15.70:0/1326546890 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15a0140a50 con 0x7f15a0066b90 2024-03-29T11:09:05.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 1 -- 172.21.15.70:0/1326546890 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 278783104 0 0) 0x7f15a0070be0 con 0x7f15a013f7a0 2024-03-29T11:09:05.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 10 monclient(hunting): _negotiate 2024-03-29T11:09:05.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:05.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:05.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 10 monclient(hunting): my global_id is 4146 2024-03-29T11:09:05.755 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 1 -- 172.21.15.70:0/1326546890 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15a0070390 con 0x7f15a013f7a0 2024-03-29T11:09:05.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 1 -- 172.21.15.70:0/1326546890 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1993181591 0 0) 0x7f158c004eb0 con 0x7f15a013f340 2024-03-29T11:09:05.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 1 -- 172.21.15.70:0/1326546890 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15a0070be0 con 0x7f15a013f340 2024-03-29T11:09:05.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 1 -- 172.21.15.70:0/1326546890 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f1594001a50 con 0x7f15a013f340 2024-03-29T11:09:05.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:05.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:09:05.756 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 10 monclient(hunting): dump: 2024-03-29T11:09:05.756 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:05.756 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:05.756 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.756 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.756 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:05.756 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:05.756 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:05.756 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:05.756 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:05.757 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:05.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:09:05.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 1 -- 172.21.15.70:0/1326546890 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3664326154 0 0) 0x7f15a0140a50 con 0x7f15a0066b90 2024-03-29T11:09:05.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 1 -- 172.21.15.70:0/1326546890 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f158c004eb0 con 0x7f15a0066b90 2024-03-29T11:09:05.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 1 -- 172.21.15.70:0/1326546890 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f1590001b80 con 0x7f15a0066b90 2024-03-29T11:09:05.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:05.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:09:05.757 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 10 monclient(hunting): dump: 2024-03-29T11:09:05.757 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:05.757 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:05.757 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.757 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.757 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:05.757 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:05.757 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:05.758 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:05.758 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:05.758 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:05.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 1 -- 172.21.15.70:0/1326546890 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2092718108 0 0) 0x7f15a0070390 con 0x7f15a013f7a0 2024-03-29T11:09:05.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 1 -- 172.21.15.70:0/1326546890 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15a0140a50 con 0x7f15a013f7a0 2024-03-29T11:09:05.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 1 -- 172.21.15.70:0/1326546890 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f1598001af0 con 0x7f15a013f7a0 2024-03-29T11:09:05.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:05.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:09:05.758 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 10 monclient(hunting): dump: 2024-03-29T11:09:05.758 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:05.758 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:05.758 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.758 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.758 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:05.759 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:05.759 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:05.759 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:05.759 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:05.759 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:05.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 1 -- 172.21.15.70:0/1326546890 <== mon.1 v1:172.21.15.70:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 263843152 0 0) 0x7f15a0070be0 con 0x7f15a013f340 2024-03-29T11:09:05.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 1 -- 172.21.15.70:0/1326546890 >> v1:172.21.15.18:6790/0 conn(0x7f15a0066b90 legacy=0x7f15a012ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:05.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 1 -- 172.21.15.70:0/1326546890 >> v1:172.21.15.18:6789/0 conn(0x7f15a013f7a0 legacy=0x7f15a01e1d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:05.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 10 monclient: _finish_hunting 0 2024-03-29T11:09:05.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 1 monclient: found mon.b 2024-03-29T11:09:05.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:05.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:05.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 1 -- 172.21.15.70:0/1326546890 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a013fe20 con 0x7f15a013f340 2024-03-29T11:09:05.759 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 10 monclient: _finish_auth 0 2024-03-29T11:09:05.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 10 monclient: _check_auth_tickets 2024-03-29T11:09:05.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:05.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f15a624a640 5 monclient: authenticate success, global_id 4144 2024-03-29T11:09:05.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f15a624a640 10 monclient: _renew_subs 2024-03-29T11:09:05.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f15a624a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:05.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f15a624a640 1 -- 172.21.15.70:0/1326546890 --> v1:172.21.15.70:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15a01352d0 con 0x7f15a013f340 2024-03-29T11:09:05.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f15a624a640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:09:05.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f15a624a640 10 monclient: _renew_subs 2024-03-29T11:09:05.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f15a624a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:05.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f15a624a640 1 -- 172.21.15.70:0/1326546890 --> v1:172.21.15.70:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f15a0071980 con 0x7f15a013f340 2024-03-29T11:09:05.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f15a624a640 1 librados: init done 2024-03-29T11:09:05.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 1 -- 172.21.15.70:0/1326546890 <== mon.1 v1:172.21.15.70:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1594001cd0 con 0x7f15a013f340 2024-03-29T11:09:05.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:05.760 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 1 -- 172.21.15.70:0/1326546890 <== mon.1 v1:172.21.15.70:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f1594004c30 con 0x7f15a013f340 2024-03-29T11:09:05.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:05.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:09:05.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f159dffb640 4 set_mon_vals no callback set 2024-03-29T11:09:05.761 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 10 monclient: dump: 2024-03-29T11:09:05.761 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:05.761 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:05.761 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.761 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.761 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:05.761 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:05.761 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:05.761 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:05.761 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:05.761 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:05.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 1 -- 172.21.15.70:0/1326546890 <== mon.1 v1:172.21.15.70:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f15940051a0 con 0x7f15a013f340 2024-03-29T11:09:05.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 1 -- 172.21.15.70:0/1326546890 <== mon.1 v1:172.21.15.70:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 10387322 0 0) 0x7f1594006930 con 0x7f15a013f340 2024-03-29T11:09:05.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 10 client.4144.objecter ms_dispatch 0x7f15a0000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T11:09:05.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 3 client.4144.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T11:09:05.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 3 client.4144.objecter handle_osd_map decoding full epoch 7 2024-03-29T11:09:05.762 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.764+0000 7f1587fff640 20 client.4144.objecter dump_active .. 0 homeless 2024-03-29T11:09:05.762 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-03-29T11:09:05.765+0000 7f6963a2c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:09:05.763 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-29T11:09:05.764+0000 7fad3de2e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:09:05.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.765+0000 7f15a624a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:05.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.765+0000 7f15a624a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:05.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.765+0000 7f15a624a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:05.763 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.765+0000 7f15a624a640 1 -- 172.21.15.70:0/1326546890 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1580005080 con 0x7f15a013f340 2024-03-29T11:09:05.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.767+0000 7f1587fff640 1 -- 172.21.15.70:0/1326546890 <== mon.1 v1:172.21.15.70:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f1594006460 con 0x7f15a013f340 2024-03-29T11:09:05.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.767+0000 7f1587fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:05.764 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.767+0000 7f1587fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:09:05.790 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-03-29T11:09:05.812+0000 7f6963a2c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:09:05.815 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.837+0000 7f15a624a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "4d29e43e-9f99-450d-9d20-6b8dbf6eaa34", "id": 5}] 2024-03-29T11:09:05.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.837+0000 7f15a624a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "4d29e43e-9f99-450d-9d20-6b8dbf6eaa34", "id": 5}] 2024-03-29T11:09:05.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.837+0000 7f15a624a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:05.816 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.837+0000 7f15a624a640 1 -- 172.21.15.70:0/1326546890 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "osd new", "uuid": "4d29e43e-9f99-450d-9d20-6b8dbf6eaa34", "id": 5} v 0) v1 -- 0x7f15800058b0 con 0x7f15a013f340 2024-03-29T11:09:05.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.839+0000 7f1587fff640 1 -- 172.21.15.70:0/1326546890 <== mon.1 v1:172.21.15.70:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4d29e43e-9f99-450d-9d20-6b8dbf6eaa34", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 1102240273 0 1988412176) 0x7f15940066d0 con 0x7f15a013f340 2024-03-29T11:09:05.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.839+0000 7f1587fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "4d29e43e-9f99-450d-9d20-6b8dbf6eaa34", "id": 5}] 2024-03-29T11:09:05.818 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.839+0000 7f1587fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:09:05.818 INFO:teuthology.orchestra.run.smithi070.stdout:5 2024-03-29T11:09:05.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.840+0000 7f15a624a640 10 librados: watch_flush enter 2024-03-29T11:09:05.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.840+0000 7f15a624a640 10 librados: watch_flush exit 2024-03-29T11:09:05.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.840+0000 7f15a624a640 20 client.4144.objecter shutdown clearing up homeless session... 2024-03-29T11:09:05.819 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.840+0000 7f15a624a640 10 client.4144.objecter successfully canceled tick 2024-03-29T11:09:05.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.840+0000 7f15a624a640 10 monclient: shutdown 2024-03-29T11:09:05.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.840+0000 7f15a624a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:05.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.840+0000 7f15a624a640 1 -- 172.21.15.70:0/1326546890 >> v1:172.21.15.70:6789/0 conn(0x7f15a013f340 legacy=0x7f15a0071090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:05.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.840+0000 7f15a624a640 1 -- 172.21.15.70:0/1326546890 shutdown_connections 2024-03-29T11:09:05.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.840+0000 7f15a624a640 1 -- 172.21.15.70:0/1326546890 >> 172.21.15.70:0/1326546890 conn(0x7f15a00b4970 msgr2=0x7f15a00b3cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:05.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.841+0000 7f15a624a640 1 -- 172.21.15.70:0/1326546890 shutdown_connections 2024-03-29T11:09:05.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.841+0000 7f15a624a640 1 -- 172.21.15.70:0/1326546890 wait complete. 2024-03-29T11:09:05.820 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.841+0000 7f15a624a640 1 librados: shutdown 2024-03-29T11:09:05.829 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --cluster ceph osd new d92168fd-c75c-43dd-b3dd-dcbf53d170d5 6 2024-03-29T11:09:05.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.948+0000 7fd1d47f4640 10 monclient: get_monmap_and_config 2024-03-29T11:09:05.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.948+0000 7fd1d47f4640 10 monclient: build_initial_monmap 2024-03-29T11:09:05.927 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.948+0000 7fd1d47f4640 10 monclient: monmap: 2024-03-29T11:09:05.927 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-03-29T11:09:05.927 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:05.927 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:09:05.949558+0000 2024-03-29T11:09:05.928 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:09:05.949558+0000 2024-03-29T11:09:05.928 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:05.928 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:05.928 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:09:05.928 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:09:05.928 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:09:05.928 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:05.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.949+0000 7fd1d47f4640 1 Processor -- start 2024-03-29T11:09:05.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.949+0000 7fd1d47f4640 1 -- start start 2024-03-29T11:09:05.928 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.949+0000 7fd1d47f4640 10 monclient: init 2024-03-29T11:09:05.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.949+0000 7fd1d47f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:05.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.949+0000 7fd1d47f4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:09:05.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.949+0000 7fd1d47f4640 10 monclient(hunting): picked mon.noname-a con 0x7fd1cc09ac40 addr v1:172.21.15.18:6789/0 2024-03-29T11:09:05.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.949+0000 7fd1d47f4640 10 monclient(hunting): picked mon.noname-c con 0x7fd1cc09c870 addr v1:172.21.15.70:6789/0 2024-03-29T11:09:05.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.949+0000 7fd1d47f4640 10 monclient(hunting): picked mon.noname-b con 0x7fd1cc0b4360 addr v1:172.21.15.18:6790/0 2024-03-29T11:09:05.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.949+0000 7fd1d47f4640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1cc09dec0 con 0x7fd1cc09ac40 2024-03-29T11:09:05.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.949+0000 7fd1d47f4640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1cc09da30 con 0x7fd1cc09c870 2024-03-29T11:09:05.937 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.949+0000 7fd1d47f4640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1cc0b39c0 con 0x7fd1cc0b4360 2024-03-29T11:09:05.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.949+0000 7fd1d47f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:05.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.949+0000 7fd1d47f4640 10 monclient(hunting): authenticate will time out at 762.178101s 2024-03-29T11:09:05.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d1d8f640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fd1cc09c870 0x7fd1cc13f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:48272/0 (socket says 172.21.15.70:48272) 2024-03-29T11:09:05.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d1d8f640 1 -- 172.21.15.70:0/3504887554 learned_addr learned my addr 172.21.15.70:0/3504887554 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-03-29T11:09:05.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d158e640 1 -- 172.21.15.70:0/3504887554 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3715124512 0 0) 0x7fd1cc09da30 con 0x7fd1cc09c870 2024-03-29T11:09:05.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d158e640 10 monclient(hunting): _negotiate 2024-03-29T11:09:05.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d158e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:05.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d158e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:05.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d158e640 10 monclient(hunting): my global_id is 4147 2024-03-29T11:09:05.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d158e640 1 -- 172.21.15.70:0/3504887554 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1b8003520 con 0x7fd1cc09c870 2024-03-29T11:09:05.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d158e640 1 -- 172.21.15.70:0/3504887554 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3755220845 0 0) 0x7fd1c8001110 con 0x7fd1cc09ac40 2024-03-29T11:09:05.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d158e640 10 monclient(hunting): _negotiate 2024-03-29T11:09:05.938 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d158e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:05.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d158e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:05.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d158e640 10 monclient(hunting): my global_id is 4149 2024-03-29T11:09:05.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d158e640 1 -- 172.21.15.70:0/3504887554 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1cc09da30 con 0x7fd1cc09ac40 2024-03-29T11:09:05.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d158e640 1 -- 172.21.15.70:0/3504887554 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 534112886 0 0) 0x7fd1cc0b39c0 con 0x7fd1cc0b4360 2024-03-29T11:09:05.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d158e640 10 monclient(hunting): _negotiate 2024-03-29T11:09:05.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d158e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:05.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d158e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:05.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d158e640 10 monclient(hunting): my global_id is 4151 2024-03-29T11:09:05.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d158e640 1 -- 172.21.15.70:0/3504887554 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1c8001110 con 0x7fd1cc0b4360 2024-03-29T11:09:05.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d158e640 1 -- 172.21.15.70:0/3504887554 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2914878120 0 0) 0x7fd1b8003520 con 0x7fd1cc09c870 2024-03-29T11:09:05.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d158e640 1 -- 172.21.15.70:0/3504887554 >> v1:172.21.15.18:6790/0 conn(0x7fd1cc0b4360 legacy=0x7fd1cc14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:05.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d158e640 1 -- 172.21.15.70:0/3504887554 >> v1:172.21.15.18:6789/0 conn(0x7fd1cc09ac40 legacy=0x7fd1cc0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:05.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d158e640 10 monclient: _finish_hunting 0 2024-03-29T11:09:05.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d158e640 1 monclient: found mon.noname-c 2024-03-29T11:09:05.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d158e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:05.939 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d158e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.70:6789/0 2024-03-29T11:09:05.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d158e640 1 -- 172.21.15.70:0/3504887554 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1cc0b5160 con 0x7fd1cc09c870 2024-03-29T11:09:05.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d158e640 10 monclient: _finish_auth 0 2024-03-29T11:09:05.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d158e640 10 monclient: _check_auth_tickets 2024-03-29T11:09:05.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d158e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:05.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d158e640 1 -- 172.21.15.70:0/3504887554 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fd1bc001690 con 0x7fd1cc09c870 2024-03-29T11:09:05.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d47f4640 5 monclient: authenticate success, global_id 4147 2024-03-29T11:09:05.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d47f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:05.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d158e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:05.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d158e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:09:05.940 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d158e640 10 monclient: dump: 2024-03-29T11:09:05.940 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:05.940 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:05.940 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.940 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.940 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:05.941 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:05.941 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:05.941 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:05.941 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:05.941 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:05.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d47f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:05.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d158e640 1 -- 172.21.15.70:0/3504887554 <== mon.1 v1:172.21.15.70:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd1bc001870 con 0x7fd1cc09c870 2024-03-29T11:09:05.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d158e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:05.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d158e640 1 -- 172.21.15.70:0/3504887554 <== mon.1 v1:172.21.15.70:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fd1bc001c80 con 0x7fd1cc09c870 2024-03-29T11:09:05.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d158e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:05.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d158e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:09:05.941 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d158e640 10 monclient: dump: 2024-03-29T11:09:05.941 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:05.941 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:05.941 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.942 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.942 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:05.942 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:05.942 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:05.942 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:05.942 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:05.942 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:05.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d47f4640 10 monclient: get_monmap_and_config success 2024-03-29T11:09:05.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.950+0000 7fd1d47f4640 4 set_mon_vals no callback set 2024-03-29T11:09:05.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.951+0000 7fd1d47f4640 10 monclient: shutdown 2024-03-29T11:09:05.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.951+0000 7fd1d47f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:05.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.951+0000 7fd1d47f4640 1 -- 172.21.15.70:0/3504887554 >> v1:172.21.15.70:6789/0 conn(0x7fd1cc09c870 legacy=0x7fd1cc13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:05.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.951+0000 7fd1d47f4640 1 -- 172.21.15.70:0/3504887554 shutdown_connections 2024-03-29T11:09:05.942 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.951+0000 7fd1d47f4640 1 -- 172.21.15.70:0/3504887554 >> 172.21.15.70:0/3504887554 conn(0x7fd1cc066b00 msgr2=0x7fd1cc066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:05.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.951+0000 7fd1d47f4640 1 -- 172.21.15.70:0/3504887554 shutdown_connections 2024-03-29T11:09:05.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.951+0000 7fd1d47f4640 1 -- 172.21.15.70:0/3504887554 wait complete. 2024-03-29T11:09:05.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.952+0000 7fd1d47f4640 10 monclient: build_initial_monmap 2024-03-29T11:09:05.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.952+0000 7fd1d47f4640 10 monclient: monmap: 2024-03-29T11:09:05.943 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-03-29T11:09:05.943 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:09:05.943 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-03-29T11:09:05.943 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-03-29T11:09:05.943 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:05.943 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:05.943 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:09:05.943 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:09:05.943 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:09:05.943 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:05.943 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.952+0000 7fd1d47f4640 1 librados: starting msgr at 2024-03-29T11:09:05.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.952+0000 7fd1d47f4640 1 librados: starting objecter 2024-03-29T11:09:05.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.953+0000 7fd1d47f4640 1 Processor -- start 2024-03-29T11:09:05.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.953+0000 7fd1d47f4640 1 -- start start 2024-03-29T11:09:05.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.953+0000 7fd1d47f4640 1 librados: setting wanted keys 2024-03-29T11:09:05.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.953+0000 7fd1d47f4640 1 librados: calling monclient init 2024-03-29T11:09:05.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.953+0000 7fd1d47f4640 10 monclient: init 2024-03-29T11:09:05.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.953+0000 7fd1d47f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:05.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.953+0000 7fd1d47f4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:09:05.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.953+0000 7fd1d47f4640 10 monclient(hunting): picked mon.noname-a con 0x7fd1cc09ac40 addr v1:172.21.15.18:6789/0 2024-03-29T11:09:05.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.953+0000 7fd1d47f4640 10 monclient(hunting): picked mon.noname-b con 0x7fd1cc0b7eb0 addr v1:172.21.15.70:6789/0 2024-03-29T11:09:05.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.953+0000 7fd1d47f4640 10 monclient(hunting): picked mon.noname-c con 0x7fd1cc13ffa0 addr v1:172.21.15.18:6790/0 2024-03-29T11:09:05.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.953+0000 7fd1d47f4640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1cc143060 con 0x7fd1cc09ac40 2024-03-29T11:09:05.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.953+0000 7fd1d47f4640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1cc1d7760 con 0x7fd1cc0b7eb0 2024-03-29T11:09:05.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.953+0000 7fd1d47f4640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1cc1d78f0 con 0x7fd1cc13ffa0 2024-03-29T11:09:05.944 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.953+0000 7fd1d47f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:05.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.953+0000 7fd1d47f4640 10 monclient(hunting): authenticate will time out at 762.182174s 2024-03-29T11:09:05.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1d1d8f640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fd1cc0b7eb0 0x7fd1cc1d7050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:48286/0 (socket says 172.21.15.70:48286) 2024-03-29T11:09:05.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1d1d8f640 1 -- 172.21.15.70:0/3023327913 learned_addr learned my addr 172.21.15.70:0/3023327913 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-03-29T11:09:05.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 10 client.?.objecter ms_handle_connect 0x7fd1cc0b7eb0 2024-03-29T11:09:05.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:05.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 10 client.?.objecter ms_handle_connect 0x7fd1cc09ac40 2024-03-29T11:09:05.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:05.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 10 client.?.objecter ms_handle_connect 0x7fd1cc13ffa0 2024-03-29T11:09:05.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:05.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 1 -- 172.21.15.70:0/3023327913 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4190225570 0 0) 0x7fd1cc1d7760 con 0x7fd1cc0b7eb0 2024-03-29T11:09:05.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:09:05.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:05.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:05.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 10 monclient(hunting): my global_id is 4150 2024-03-29T11:09:05.945 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 1 -- 172.21.15.70:0/3023327913 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1b0004eb0 con 0x7fd1cc0b7eb0 2024-03-29T11:09:05.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 1 -- 172.21.15.70:0/3023327913 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3583452429 0 0) 0x7fd1cc143060 con 0x7fd1cc09ac40 2024-03-29T11:09:05.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:09:05.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:05.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:05.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 10 monclient(hunting): my global_id is 4152 2024-03-29T11:09:05.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 1 -- 172.21.15.70:0/3023327913 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1cc1d7760 con 0x7fd1cc09ac40 2024-03-29T11:09:05.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 1 -- 172.21.15.70:0/3023327913 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 782793621 0 0) 0x7fd1cc1d78f0 con 0x7fd1cc13ffa0 2024-03-29T11:09:05.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:09:05.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:05.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:05.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 10 monclient(hunting): my global_id is 4154 2024-03-29T11:09:05.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 1 -- 172.21.15.70:0/3023327913 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1cc143060 con 0x7fd1cc13ffa0 2024-03-29T11:09:05.946 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 1 -- 172.21.15.70:0/3023327913 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 890810544 0 0) 0x7fd1b0004eb0 con 0x7fd1cc0b7eb0 2024-03-29T11:09:05.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 1 -- 172.21.15.70:0/3023327913 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1cc1d78f0 con 0x7fd1cc0b7eb0 2024-03-29T11:09:05.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 1 -- 172.21.15.70:0/3023327913 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fd1bc001bc0 con 0x7fd1cc0b7eb0 2024-03-29T11:09:05.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:05.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:09:05.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 10 monclient(hunting): dump: 2024-03-29T11:09:05.947 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:05.947 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:05.947 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.947 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.947 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:05.947 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:05.947 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:05.947 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:05.947 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:05.947 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:05.947 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:09:05.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 1 -- 172.21.15.70:0/3023327913 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1153925665 0 0) 0x7fd1cc1d7760 con 0x7fd1cc09ac40 2024-03-29T11:09:05.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 1 -- 172.21.15.70:0/3023327913 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1b0004eb0 con 0x7fd1cc09ac40 2024-03-29T11:09:05.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 1 -- 172.21.15.70:0/3023327913 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 929299769 0 0) 0x7fd1cc143060 con 0x7fd1cc13ffa0 2024-03-29T11:09:05.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 1 -- 172.21.15.70:0/3023327913 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1cc1d7760 con 0x7fd1cc13ffa0 2024-03-29T11:09:05.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 1 -- 172.21.15.70:0/3023327913 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fd1c80018a0 con 0x7fd1cc09ac40 2024-03-29T11:09:05.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:05.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:09:05.948 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 10 monclient(hunting): dump: 2024-03-29T11:09:05.948 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:05.948 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:05.948 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.948 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.948 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:05.948 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:05.948 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:05.948 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:05.949 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:05.949 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:05.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 1 -- 172.21.15.70:0/3023327913 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fd1c4001b50 con 0x7fd1cc13ffa0 2024-03-29T11:09:05.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:05.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:09:05.949 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 10 monclient(hunting): dump: 2024-03-29T11:09:05.949 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:05.949 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:05.949 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.949 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.949 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:05.949 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:05.949 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:05.949 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:05.949 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:05.950 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:05.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 1 -- 172.21.15.70:0/3023327913 <== mon.1 v1:172.21.15.70:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2936709671 0 0) 0x7fd1cc1d78f0 con 0x7fd1cc0b7eb0 2024-03-29T11:09:05.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 1 -- 172.21.15.70:0/3023327913 >> v1:172.21.15.18:6790/0 conn(0x7fd1cc13ffa0 legacy=0x7fd1cc1e1a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:05.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 1 -- 172.21.15.70:0/3023327913 >> v1:172.21.15.18:6789/0 conn(0x7fd1cc09ac40 legacy=0x7fd1cc1427d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:05.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:09:05.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 1 monclient: found mon.b 2024-03-29T11:09:05.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:05.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:05.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 1 -- 172.21.15.70:0/3023327913 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1cc140460 con 0x7fd1cc0b7eb0 2024-03-29T11:09:05.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:09:05.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:09:05.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:05.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1d47f4640 5 monclient: authenticate success, global_id 4150 2024-03-29T11:09:05.950 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1d47f4640 10 monclient: _renew_subs 2024-03-29T11:09:05.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1d47f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:05.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1d47f4640 1 -- 172.21.15.70:0/3023327913 --> v1:172.21.15.70:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1cc140710 con 0x7fd1cc0b7eb0 2024-03-29T11:09:05.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1d47f4640 10 client.4150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:09:05.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1d47f4640 10 monclient: _renew_subs 2024-03-29T11:09:05.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1d47f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:05.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1d47f4640 1 -- 172.21.15.70:0/3023327913 --> v1:172.21.15.70:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1cc1e3390 con 0x7fd1cc0b7eb0 2024-03-29T11:09:05.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1d47f4640 1 librados: init done 2024-03-29T11:09:05.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 1 -- 172.21.15.70:0/3023327913 <== mon.1 v1:172.21.15.70:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd1bc004bf0 con 0x7fd1cc0b7eb0 2024-03-29T11:09:05.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:05.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 1 -- 172.21.15.70:0/3023327913 <== mon.1 v1:172.21.15.70:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fd1bc004f00 con 0x7fd1cc0b7eb0 2024-03-29T11:09:05.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:05.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:09:05.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1d0d8d640 4 set_mon_vals no callback set 2024-03-29T11:09:05.951 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 10 monclient: dump: 2024-03-29T11:09:05.951 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:05.952 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:05.952 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.952 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:05.952 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:05.952 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:05.952 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:05.952 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:05.952 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:05.952 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:05.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 1 -- 172.21.15.70:0/3023327913 <== mon.1 v1:172.21.15.70:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fd1bc0053f0 con 0x7fd1cc0b7eb0 2024-03-29T11:09:05.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 1 -- 172.21.15.70:0/3023327913 <== mon.1 v1:172.21.15.70:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 3756003253 0 0) 0x7fd1bc006ad0 con 0x7fd1cc0b7eb0 2024-03-29T11:09:05.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 10 client.4150.objecter ms_dispatch 0x7fd1cc000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T11:09:05.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 3 client.4150.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T11:09:05.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.954+0000 7fd1c2ffd640 3 client.4150.objecter handle_osd_map decoding full epoch 8 2024-03-29T11:09:05.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.955+0000 7fd1c2ffd640 20 client.4150.objecter dump_active .. 0 homeless 2024-03-29T11:09:05.952 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.956+0000 7fd1d47f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:05.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.956+0000 7fd1d47f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:05.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.956+0000 7fd1d47f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:05.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.956+0000 7fd1d47f4640 1 -- 172.21.15.70:0/3023327913 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1b4005080 con 0x7fd1cc0b7eb0 2024-03-29T11:09:05.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.958+0000 7fd1c2ffd640 1 -- 172.21.15.70:0/3023327913 <== mon.1 v1:172.21.15.70:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fd1bc0064e0 con 0x7fd1cc0b7eb0 2024-03-29T11:09:05.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.958+0000 7fd1c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:05.953 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:05.958+0000 7fd1c2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:09:06.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.050+0000 7fd1d47f4640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d92168fd-c75c-43dd-b3dd-dcbf53d170d5", "id": 6}] 2024-03-29T11:09:06.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.050+0000 7fd1d47f4640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d92168fd-c75c-43dd-b3dd-dcbf53d170d5", "id": 6}] 2024-03-29T11:09:06.028 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.050+0000 7fd1d47f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:06.029 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.050+0000 7fd1d47f4640 1 -- 172.21.15.70:0/3023327913 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "osd new", "uuid": "d92168fd-c75c-43dd-b3dd-dcbf53d170d5", "id": 6} v 0) v1 -- 0x7fd1b40058b0 con 0x7fd1cc0b7eb0 2024-03-29T11:09:06.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.052+0000 7fd1c2ffd640 1 -- 172.21.15.70:0/3023327913 <== mon.1 v1:172.21.15.70:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d92168fd-c75c-43dd-b3dd-dcbf53d170d5", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 636436183 0 1113811849) 0x7fd1bc006810 con 0x7fd1cc0b7eb0 2024-03-29T11:09:06.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.052+0000 7fd1c2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d92168fd-c75c-43dd-b3dd-dcbf53d170d5", "id": 6}] 2024-03-29T11:09:06.031 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.052+0000 7fd1c2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:09:06.031 INFO:teuthology.orchestra.run.smithi070.stdout:6 2024-03-29T11:09:06.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.053+0000 7fd1d47f4640 10 librados: watch_flush enter 2024-03-29T11:09:06.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.053+0000 7fd1d47f4640 10 librados: watch_flush exit 2024-03-29T11:09:06.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.053+0000 7fd1d47f4640 20 client.4150.objecter shutdown clearing up homeless session... 2024-03-29T11:09:06.032 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.053+0000 7fd1d47f4640 10 client.4150.objecter successfully canceled tick 2024-03-29T11:09:06.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.053+0000 7fd1d47f4640 10 monclient: shutdown 2024-03-29T11:09:06.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.053+0000 7fd1d47f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:06.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.053+0000 7fd1d47f4640 1 -- 172.21.15.70:0/3023327913 >> v1:172.21.15.70:6789/0 conn(0x7fd1cc0b7eb0 legacy=0x7fd1cc1d7050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:06.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.053+0000 7fd1d47f4640 1 -- 172.21.15.70:0/3023327913 shutdown_connections 2024-03-29T11:09:06.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.053+0000 7fd1d47f4640 1 -- 172.21.15.70:0/3023327913 >> 172.21.15.70:0/3023327913 conn(0x7fd1cc066b00 msgr2=0x7fd1cc13d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:06.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.053+0000 7fd1d47f4640 1 -- 172.21.15.70:0/3023327913 shutdown_connections 2024-03-29T11:09:06.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.053+0000 7fd1d47f4640 1 -- 172.21.15.70:0/3023327913 wait complete. 2024-03-29T11:09:06.033 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.053+0000 7fd1d47f4640 1 librados: shutdown 2024-03-29T11:09:06.042 DEBUG:teuthology.orchestra.run.smithi070:> sudo ceph --cluster ceph osd new 89acf17e-ff8f-40ad-bdcc-da3d92fb9738 7 2024-03-29T11:09:06.141 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.162+0000 7fb32f3c1640 10 monclient: get_monmap_and_config 2024-03-29T11:09:06.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.162+0000 7fb32f3c1640 10 monclient: build_initial_monmap 2024-03-29T11:09:06.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.162+0000 7fb32f3c1640 10 monclient: monmap: 2024-03-29T11:09:06.142 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-03-29T11:09:06.142 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:06.142 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:09:06.163975+0000 2024-03-29T11:09:06.142 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:09:06.163975+0000 2024-03-29T11:09:06.142 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:06.142 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:06.142 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:09:06.142 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:09:06.142 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:09:06.142 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:06.142 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.163+0000 7fb32f3c1640 1 Processor -- start 2024-03-29T11:09:06.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.163+0000 7fb32f3c1640 1 -- start start 2024-03-29T11:09:06.143 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.163+0000 7fb32f3c1640 10 monclient: init 2024-03-29T11:09:06.143 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-29T11:09:06.163+0000 7fad3de2e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:09:06.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb32f3c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:06.151 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb32f3c1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:09:06.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb32f3c1640 10 monclient(hunting): picked mon.noname-a con 0x7fb3280b1780 addr v1:172.21.15.18:6789/0 2024-03-29T11:09:06.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb32f3c1640 10 monclient(hunting): picked mon.noname-c con 0x7fb32813d7e0 addr v1:172.21.15.70:6789/0 2024-03-29T11:09:06.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb32f3c1640 10 monclient(hunting): picked mon.noname-b con 0x7fb328143be0 addr v1:172.21.15.18:6790/0 2024-03-29T11:09:06.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb32f3c1640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb328070b30 con 0x7fb3280b1780 2024-03-29T11:09:06.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb32f3c1640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb328070da0 con 0x7fb32813d7e0 2024-03-29T11:09:06.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb32f3c1640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb32809adb0 con 0x7fb328143be0 2024-03-29T11:09:06.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb32f3c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:06.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb32f3c1640 10 monclient(hunting): authenticate will time out at 762.392546s 2024-03-29T11:09:06.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb32c95c640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fb32813d7e0 0x7fb3281434d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:48298/0 (socket says 172.21.15.70:48298) 2024-03-29T11:09:06.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb32c95c640 1 -- 172.21.15.70:0/1333278095 learned_addr learned my addr 172.21.15.70:0/1333278095 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-03-29T11:09:06.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb31ffff640 1 -- 172.21.15.70:0/1333278095 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2471847428 0 0) 0x7fb328070da0 con 0x7fb32813d7e0 2024-03-29T11:09:06.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb31ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:09:06.152 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb31ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:06.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb31ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:06.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb31ffff640 10 monclient(hunting): my global_id is 4153 2024-03-29T11:09:06.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb31ffff640 1 -- 172.21.15.70:0/1333278095 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb314003520 con 0x7fb32813d7e0 2024-03-29T11:09:06.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb31ffff640 1 -- 172.21.15.70:0/1333278095 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2489518720 0 0) 0x7fb328070b30 con 0x7fb3280b1780 2024-03-29T11:09:06.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb31ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:09:06.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb31ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:06.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb31ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:06.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb31ffff640 10 monclient(hunting): my global_id is 4155 2024-03-29T11:09:06.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb31ffff640 1 -- 172.21.15.70:0/1333278095 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb328070da0 con 0x7fb3280b1780 2024-03-29T11:09:06.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb31ffff640 1 -- 172.21.15.70:0/1333278095 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 96027160 0 0) 0x7fb32809adb0 con 0x7fb328143be0 2024-03-29T11:09:06.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb31ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:09:06.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb31ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:06.153 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb31ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:06.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb31ffff640 10 monclient(hunting): my global_id is 4157 2024-03-29T11:09:06.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb31ffff640 1 -- 172.21.15.70:0/1333278095 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb328070b30 con 0x7fb328143be0 2024-03-29T11:09:06.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb31ffff640 1 -- 172.21.15.70:0/1333278095 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3227186753 0 0) 0x7fb314003520 con 0x7fb32813d7e0 2024-03-29T11:09:06.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb31ffff640 1 -- 172.21.15.70:0/1333278095 >> v1:172.21.15.18:6790/0 conn(0x7fb328143be0 legacy=0x7fb32814b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:06.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb31ffff640 1 -- 172.21.15.70:0/1333278095 >> v1:172.21.15.18:6789/0 conn(0x7fb3280b1780 legacy=0x7fb32813d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:06.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb31ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:09:06.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb31ffff640 1 monclient: found mon.noname-c 2024-03-29T11:09:06.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb31ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:06.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb31ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.70:6789/0 2024-03-29T11:09:06.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb31ffff640 1 -- 172.21.15.70:0/1333278095 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb328144070 con 0x7fb32813d7e0 2024-03-29T11:09:06.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb31ffff640 10 monclient: _finish_auth 0 2024-03-29T11:09:06.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb31ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:09:06.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb31ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:06.154 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb31ffff640 1 -- 172.21.15.70:0/1333278095 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fb324001690 con 0x7fb32813d7e0 2024-03-29T11:09:06.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb32f3c1640 5 monclient: authenticate success, global_id 4153 2024-03-29T11:09:06.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb32f3c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:06.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb31ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:06.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb31ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:09:06.155 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb31ffff640 10 monclient: dump: 2024-03-29T11:09:06.155 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:06.155 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:06.155 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:06.156 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:06.156 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:06.156 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:06.156 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:06.156 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:06.156 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:06.157 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:06.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.164+0000 7fb32f3c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:06.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.165+0000 7fb31ffff640 1 -- 172.21.15.70:0/1333278095 <== mon.1 v1:172.21.15.70:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb324001870 con 0x7fb32813d7e0 2024-03-29T11:09:06.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.165+0000 7fb31ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:06.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.165+0000 7fb31ffff640 1 -- 172.21.15.70:0/1333278095 <== mon.1 v1:172.21.15.70:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fb324001c80 con 0x7fb32813d7e0 2024-03-29T11:09:06.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.165+0000 7fb32f3c1640 10 monclient: get_monmap_and_config success 2024-03-29T11:09:06.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.165+0000 7fb32f3c1640 4 set_mon_vals no callback set 2024-03-29T11:09:06.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.165+0000 7fb31ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:06.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.165+0000 7fb31ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:09:06.157 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.165+0000 7fb31ffff640 10 monclient: dump: 2024-03-29T11:09:06.157 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:06.157 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:06.157 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:06.157 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:06.158 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:06.158 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:06.158 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:06.158 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:06.158 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:06.158 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:06.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.165+0000 7fb32f3c1640 10 monclient: shutdown 2024-03-29T11:09:06.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.165+0000 7fb32f3c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:06.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.165+0000 7fb32f3c1640 1 -- 172.21.15.70:0/1333278095 >> v1:172.21.15.70:6789/0 conn(0x7fb32813d7e0 legacy=0x7fb3281434d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:06.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.165+0000 7fb32f3c1640 1 -- 172.21.15.70:0/1333278095 shutdown_connections 2024-03-29T11:09:06.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.165+0000 7fb32f3c1640 1 -- 172.21.15.70:0/1333278095 >> 172.21.15.70:0/1333278095 conn(0x7fb3280b58f0 msgr2=0x7fb3280b5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:06.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.165+0000 7fb32f3c1640 1 -- 172.21.15.70:0/1333278095 shutdown_connections 2024-03-29T11:09:06.158 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.165+0000 7fb32f3c1640 1 -- 172.21.15.70:0/1333278095 wait complete. 2024-03-29T11:09:06.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.166+0000 7fb32f3c1640 10 monclient: build_initial_monmap 2024-03-29T11:09:06.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.166+0000 7fb32f3c1640 10 monclient: monmap: 2024-03-29T11:09:06.159 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 0 2024-03-29T11:09:06.159 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:09:06.159 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 0.000000 2024-03-29T11:09:06.159 INFO:teuthology.orchestra.run.smithi070.stderr:created 0.000000 2024-03-29T11:09:06.159 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:06.159 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:06.159 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:09:06.159 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:09:06.159 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:09:06.159 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:06.159 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.166+0000 7fb32f3c1640 1 librados: starting msgr at 2024-03-29T11:09:06.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.166+0000 7fb32f3c1640 1 librados: starting objecter 2024-03-29T11:09:06.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.166+0000 7fb32f3c1640 1 Processor -- start 2024-03-29T11:09:06.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.166+0000 7fb32f3c1640 1 -- start start 2024-03-29T11:09:06.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.166+0000 7fb32f3c1640 1 librados: setting wanted keys 2024-03-29T11:09:06.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.166+0000 7fb32f3c1640 1 librados: calling monclient init 2024-03-29T11:09:06.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.166+0000 7fb32f3c1640 10 monclient: init 2024-03-29T11:09:06.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb32f3c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:06.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb32f3c1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:09:06.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb32f3c1640 10 monclient(hunting): picked mon.noname-c con 0x7fb3280b1780 addr v1:172.21.15.18:6790/0 2024-03-29T11:09:06.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb32f3c1640 10 monclient(hunting): picked mon.noname-a con 0x7fb32813d7e0 addr v1:172.21.15.18:6789/0 2024-03-29T11:09:06.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb32f3c1640 10 monclient(hunting): picked mon.noname-b con 0x7fb328143be0 addr v1:172.21.15.70:6789/0 2024-03-29T11:09:06.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb32f3c1640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb328139db0 con 0x7fb32813d7e0 2024-03-29T11:09:06.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb32f3c1640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb328138f60 con 0x7fb328143be0 2024-03-29T11:09:06.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb32f3c1640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb32813dd40 con 0x7fb3280b1780 2024-03-29T11:09:06.160 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb32f3c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:06.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb32f3c1640 10 monclient(hunting): authenticate will time out at 762.395464s 2024-03-29T11:09:06.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb32d95e640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fb328143be0 0x7fb328140330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.70:48314/0 (socket says 172.21.15.70:48314) 2024-03-29T11:09:06.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb32d95e640 1 -- 172.21.15.70:0/2178457210 learned_addr learned my addr 172.21.15.70:0/2178457210 (peer_addr_for_me v1:172.21.15.70:0/0) 2024-03-29T11:09:06.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 10 client.?.objecter ms_handle_connect 0x7fb328143be0 2024-03-29T11:09:06.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:06.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 10 client.?.objecter ms_handle_connect 0x7fb3280b1780 2024-03-29T11:09:06.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:06.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 10 client.?.objecter ms_handle_connect 0x7fb32813d7e0 2024-03-29T11:09:06.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:06.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 1 -- 172.21.15.70:0/2178457210 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4262492836 0 0) 0x7fb328138f60 con 0x7fb328143be0 2024-03-29T11:09:06.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:09:06.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:06.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:06.161 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 10 monclient(hunting): my global_id is 4156 2024-03-29T11:09:06.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 1 -- 172.21.15.70:0/2178457210 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb30c004eb0 con 0x7fb328143be0 2024-03-29T11:09:06.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 1 -- 172.21.15.70:0/2178457210 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2213480613 0 0) 0x7fb328139db0 con 0x7fb32813d7e0 2024-03-29T11:09:06.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:09:06.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:06.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:06.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 10 monclient(hunting): my global_id is 4158 2024-03-29T11:09:06.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 1 -- 172.21.15.70:0/2178457210 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb328138f60 con 0x7fb32813d7e0 2024-03-29T11:09:06.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 1 -- 172.21.15.70:0/2178457210 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3031840177 0 0) 0x7fb32813dd40 con 0x7fb3280b1780 2024-03-29T11:09:06.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:09:06.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:06.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:06.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 10 monclient(hunting): my global_id is 4160 2024-03-29T11:09:06.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 1 -- 172.21.15.70:0/2178457210 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb328139db0 con 0x7fb3280b1780 2024-03-29T11:09:06.162 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 1 -- 172.21.15.70:0/2178457210 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3016523427 0 0) 0x7fb30c004eb0 con 0x7fb328143be0 2024-03-29T11:09:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 1 -- 172.21.15.70:0/2178457210 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb32813dd40 con 0x7fb328143be0 2024-03-29T11:09:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 1 -- 172.21.15.70:0/2178457210 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fb320001c70 con 0x7fb328143be0 2024-03-29T11:09:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:09:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 10 monclient(hunting): dump: 2024-03-29T11:09:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:06.163 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:06.163 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:06.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:09:06.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 1 -- 172.21.15.70:0/2178457210 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2530721480 0 0) 0x7fb328138f60 con 0x7fb32813d7e0 2024-03-29T11:09:06.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 1 -- 172.21.15.70:0/2178457210 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb30c004eb0 con 0x7fb32813d7e0 2024-03-29T11:09:06.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 1 -- 172.21.15.70:0/2178457210 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fb324001b80 con 0x7fb32813d7e0 2024-03-29T11:09:06.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:06.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:09:06.164 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 10 monclient(hunting): dump: 2024-03-29T11:09:06.164 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:06.164 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:06.164 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:06.164 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:06.164 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:06.164 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:06.164 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:06.164 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:06.164 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:06.165 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:06.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 1 -- 172.21.15.70:0/2178457210 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3487584837 0 0) 0x7fb328139db0 con 0x7fb3280b1780 2024-03-29T11:09:06.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 1 -- 172.21.15.70:0/2178457210 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb328138f60 con 0x7fb3280b1780 2024-03-29T11:09:06.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 1 -- 172.21.15.70:0/2178457210 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fb318001870 con 0x7fb3280b1780 2024-03-29T11:09:06.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:06.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:09:06.165 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.167+0000 7fb31dffb640 10 monclient(hunting): dump: 2024-03-29T11:09:06.165 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:06.165 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:06.165 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:06.165 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:06.165 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:06.165 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:06.165 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:06.166 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:06.166 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:06.166 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:06.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.168+0000 7fb31dffb640 1 -- 172.21.15.70:0/2178457210 <== mon.1 v1:172.21.15.70:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3747738457 0 0) 0x7fb32813dd40 con 0x7fb328143be0 2024-03-29T11:09:06.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.168+0000 7fb31dffb640 1 -- 172.21.15.70:0/2178457210 >> v1:172.21.15.18:6790/0 conn(0x7fb3280b1780 legacy=0x7fb3281429c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:06.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.168+0000 7fb31dffb640 1 -- 172.21.15.70:0/2178457210 >> v1:172.21.15.18:6789/0 conn(0x7fb32813d7e0 legacy=0x7fb3281d84d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:06.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.168+0000 7fb31dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:09:06.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.168+0000 7fb31dffb640 1 monclient: found mon.b 2024-03-29T11:09:06.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.168+0000 7fb31dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:06.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.168+0000 7fb31dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:06.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.168+0000 7fb31dffb640 1 -- 172.21.15.70:0/2178457210 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3281336a0 con 0x7fb328143be0 2024-03-29T11:09:06.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.168+0000 7fb31dffb640 10 monclient: _finish_auth 0 2024-03-29T11:09:06.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.168+0000 7fb31dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:09:06.166 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.168+0000 7fb31dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:06.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.168+0000 7fb32f3c1640 5 monclient: authenticate success, global_id 4156 2024-03-29T11:09:06.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.168+0000 7fb32f3c1640 10 monclient: _renew_subs 2024-03-29T11:09:06.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.168+0000 7fb32f3c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:06.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.168+0000 7fb32f3c1640 1 -- 172.21.15.70:0/2178457210 --> v1:172.21.15.70:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb328139940 con 0x7fb328143be0 2024-03-29T11:09:06.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.168+0000 7fb31dffb640 1 -- 172.21.15.70:0/2178457210 <== mon.1 v1:172.21.15.70:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb320003990 con 0x7fb328143be0 2024-03-29T11:09:06.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.168+0000 7fb31dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:06.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.168+0000 7fb31dffb640 1 -- 172.21.15.70:0/2178457210 <== mon.1 v1:172.21.15.70:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fb320003da0 con 0x7fb328143be0 2024-03-29T11:09:06.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.168+0000 7fb31dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:06.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.168+0000 7fb31dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:09:06.167 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.168+0000 7fb31dffb640 10 monclient: dump: 2024-03-29T11:09:06.167 INFO:teuthology.orchestra.run.smithi070.stderr:epoch 1 2024-03-29T11:09:06.167 INFO:teuthology.orchestra.run.smithi070.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:06.167 INFO:teuthology.orchestra.run.smithi070.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:06.167 INFO:teuthology.orchestra.run.smithi070.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:06.167 INFO:teuthology.orchestra.run.smithi070.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:06.168 INFO:teuthology.orchestra.run.smithi070.stderr:election_strategy: 1 2024-03-29T11:09:06.168 INFO:teuthology.orchestra.run.smithi070.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:06.168 INFO:teuthology.orchestra.run.smithi070.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:06.168 INFO:teuthology.orchestra.run.smithi070.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:06.168 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:09:06.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.168+0000 7fb31dffb640 1 -- 172.21.15.70:0/2178457210 <== mon.1 v1:172.21.15.70:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fb320004310 con 0x7fb328143be0 2024-03-29T11:09:06.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.168+0000 7fb31f7fe640 4 set_mon_vals no callback set 2024-03-29T11:09:06.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.168+0000 7fb32f3c1640 10 client.4156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:09:06.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.168+0000 7fb32f3c1640 10 monclient: _renew_subs 2024-03-29T11:09:06.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.168+0000 7fb32f3c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:06.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.168+0000 7fb32f3c1640 1 -- 172.21.15.70:0/2178457210 --> v1:172.21.15.70:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3281d8dc0 con 0x7fb328143be0 2024-03-29T11:09:06.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.168+0000 7fb32f3c1640 1 librados: init done 2024-03-29T11:09:06.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.168+0000 7fb31dffb640 1 -- 172.21.15.70:0/2178457210 <== mon.1 v1:172.21.15.70:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 3792132968 0 0) 0x7fb320006600 con 0x7fb328143be0 2024-03-29T11:09:06.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.168+0000 7fb31dffb640 10 client.4156.objecter ms_dispatch 0x7fb328000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T11:09:06.168 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.168+0000 7fb31dffb640 3 client.4156.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T11:09:06.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.168+0000 7fb31dffb640 3 client.4156.objecter handle_osd_map decoding full epoch 9 2024-03-29T11:09:06.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.168+0000 7fb31dffb640 20 client.4156.objecter dump_active .. 0 homeless 2024-03-29T11:09:06.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.169+0000 7fb32f3c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:06.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.169+0000 7fb32f3c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:06.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.169+0000 7fb32f3c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:06.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.169+0000 7fb32f3c1640 1 -- 172.21.15.70:0/2178457210 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb328067c00 con 0x7fb328143be0 2024-03-29T11:09:06.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.171+0000 7fb31dffb640 1 -- 172.21.15.70:0/2178457210 <== mon.1 v1:172.21.15.70:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fb3281d8dc0 con 0x7fb328143be0 2024-03-29T11:09:06.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.171+0000 7fb31dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:06.169 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.171+0000 7fb31dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:09:06.202 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-03-29T11:09:06.224+0000 7f6963a2c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:09:06.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.241+0000 7fb32f3c1640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "89acf17e-ff8f-40ad-bdcc-da3d92fb9738", "id": 7}] 2024-03-29T11:09:06.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.241+0000 7fb32f3c1640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "89acf17e-ff8f-40ad-bdcc-da3d92fb9738", "id": 7}] 2024-03-29T11:09:06.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.241+0000 7fb32f3c1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:06.219 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.241+0000 7fb32f3c1640 1 -- 172.21.15.70:0/2178457210 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "osd new", "uuid": "89acf17e-ff8f-40ad-bdcc-da3d92fb9738", "id": 7} v 0) v1 -- 0x7fb32804f730 con 0x7fb328143be0 2024-03-29T11:09:06.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.243+0000 7fb31dffb640 1 -- 172.21.15.70:0/2178457210 <== mon.1 v1:172.21.15.70:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "89acf17e-ff8f-40ad-bdcc-da3d92fb9738", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 1191102823 0 1371665406) 0x7fb3200047f0 con 0x7fb328143be0 2024-03-29T11:09:06.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.243+0000 7fb31dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "89acf17e-ff8f-40ad-bdcc-da3d92fb9738", "id": 7}] 2024-03-29T11:09:06.222 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.243+0000 7fb31dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:09:06.222 INFO:teuthology.orchestra.run.smithi070.stdout:7 2024-03-29T11:09:06.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.244+0000 7fb32f3c1640 10 librados: watch_flush enter 2024-03-29T11:09:06.223 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.244+0000 7fb32f3c1640 10 librados: watch_flush exit 2024-03-29T11:09:06.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.244+0000 7fb32f3c1640 20 client.4156.objecter shutdown clearing up homeless session... 2024-03-29T11:09:06.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.244+0000 7fb32f3c1640 10 client.4156.objecter successfully canceled tick 2024-03-29T11:09:06.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.244+0000 7fb32f3c1640 10 monclient: shutdown 2024-03-29T11:09:06.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.244+0000 7fb32f3c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:06.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.244+0000 7fb32f3c1640 1 -- 172.21.15.70:0/2178457210 >> v1:172.21.15.70:6789/0 conn(0x7fb328143be0 legacy=0x7fb328140330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:06.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.244+0000 7fb32f3c1640 1 -- 172.21.15.70:0/2178457210 shutdown_connections 2024-03-29T11:09:06.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.244+0000 7fb32f3c1640 1 -- 172.21.15.70:0/2178457210 >> 172.21.15.70:0/2178457210 conn(0x7fb3280b58f0 msgr2=0x7fb328069bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:06.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.244+0000 7fb32f3c1640 1 -- 172.21.15.70:0/2178457210 shutdown_connections 2024-03-29T11:09:06.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.244+0000 7fb32f3c1640 1 -- 172.21.15.70:0/2178457210 wait complete. 2024-03-29T11:09:06.224 INFO:teuthology.orchestra.run.smithi070.stderr:2024-03-29T11:09:06.244+0000 7fb32f3c1640 1 librados: shutdown 2024-03-29T11:09:06.233 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T11:09:06.233 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T11:09:06.235 INFO:tasks.ceph.osd.0:Started 2024-03-29T11:09:06.235 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T11:09:06.236 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T11:09:06.238 INFO:tasks.ceph.osd.1:Started 2024-03-29T11:09:06.238 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T11:09:06.238 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T11:09:06.241 INFO:tasks.ceph.osd.2:Started 2024-03-29T11:09:06.242 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T11:09:06.242 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T11:09:06.246 INFO:tasks.ceph.osd.3:Started 2024-03-29T11:09:06.246 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T11:09:06.246 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T11:09:06.249 INFO:tasks.ceph.osd.4:Started 2024-03-29T11:09:06.249 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T11:09:06.249 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T11:09:06.252 INFO:tasks.ceph.osd.5:Started 2024-03-29T11:09:06.252 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T11:09:06.252 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T11:09:06.254 INFO:tasks.ceph.osd.6:Started 2024-03-29T11:09:06.254 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T11:09:06.254 DEBUG:teuthology.orchestra.run.smithi070:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T11:09:06.257 INFO:tasks.ceph.osd.7:Started 2024-03-29T11:09:06.257 DEBUG:teuthology.orchestra.run.smithi018:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:09:06.294 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-29T11:09:06.316+0000 7fad3de2e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:09:06.363 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-03-29T11:09:06.385+0000 7f6963a2c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:09:06.399 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.421+0000 7fa626f04640 10 monclient: get_monmap_and_config 2024-03-29T11:09:06.399 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.421+0000 7fa626f04640 10 monclient: build_initial_monmap 2024-03-29T11:09:06.399 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.421+0000 7fa626f04640 10 monclient: monmap: 2024-03-29T11:09:06.399 INFO:teuthology.orchestra.run.smithi018.stderr:epoch 0 2024-03-29T11:09:06.399 INFO:teuthology.orchestra.run.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:06.399 INFO:teuthology.orchestra.run.smithi018.stderr:last_changed 2024-03-29T11:09:06.422012+0000 2024-03-29T11:09:06.399 INFO:teuthology.orchestra.run.smithi018.stderr:created 2024-03-29T11:09:06.422012+0000 2024-03-29T11:09:06.400 INFO:teuthology.orchestra.run.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:06.400 INFO:teuthology.orchestra.run.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:06.400 INFO:teuthology.orchestra.run.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:09:06.400 INFO:teuthology.orchestra.run.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:09:06.400 INFO:teuthology.orchestra.run.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:09:06.400 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-29T11:09:06.408 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.421+0000 7fa626f04640 1 Processor -- start 2024-03-29T11:09:06.408 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.421+0000 7fa626f04640 1 -- start start 2024-03-29T11:09:06.408 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.421+0000 7fa626f04640 10 monclient: init 2024-03-29T11:09:06.408 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.422+0000 7fa626f04640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:06.408 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.422+0000 7fa626f04640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:09:06.408 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.422+0000 7fa626f04640 10 monclient(hunting): picked mon.noname-a con 0x7fa62009ac40 addr v1:172.21.15.18:6789/0 2024-03-29T11:09:06.409 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.422+0000 7fa626f04640 10 monclient(hunting): picked mon.noname-c con 0x7fa62009c870 addr v1:172.21.15.70:6789/0 2024-03-29T11:09:06.409 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.422+0000 7fa626f04640 10 monclient(hunting): picked mon.noname-b con 0x7fa6200b4360 addr v1:172.21.15.18:6790/0 2024-03-29T11:09:06.409 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.422+0000 7fa626f04640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa62009dec0 con 0x7fa62009ac40 2024-03-29T11:09:06.409 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.422+0000 7fa626f04640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa62009da30 con 0x7fa62009c870 2024-03-29T11:09:06.409 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.422+0000 7fa626f04640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6200b39c0 con 0x7fa6200b4360 2024-03-29T11:09:06.409 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.422+0000 7fa626f04640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:06.409 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.422+0000 7fa626f04640 10 monclient(hunting): authenticate will time out at 772.694480s 2024-03-29T11:09:06.409 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.422+0000 7fa624ca0640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fa62009ac40 0x7fa6200b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:54952/0 (socket says 172.21.15.18:54952) 2024-03-29T11:09:06.409 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.422+0000 7fa624ca0640 1 -- 172.21.15.18:0/1246975217 learned_addr learned my addr 172.21.15.18:0/1246975217 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:09:06.409 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.422+0000 7fa6254a1640 1 --1- 172.21.15.18:0/1246975217 >> v1:172.21.15.18:6790/0 conn(0x7fa6200b4360 0x7fa62014d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:51562/0 (socket says 172.21.15.18:51562) 2024-03-29T11:09:06.409 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.422+0000 7fa61f7fe640 1 -- 172.21.15.18:0/1246975217 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 254781414 0 0) 0x7fa6200b39c0 con 0x7fa6200b4360 2024-03-29T11:09:06.409 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.422+0000 7fa61f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:09:06.409 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.422+0000 7fa61f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:06.409 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.422+0000 7fa61f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:06.410 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.422+0000 7fa61f7fe640 10 monclient(hunting): my global_id is 4187 2024-03-29T11:09:06.410 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.422+0000 7fa61f7fe640 1 -- 172.21.15.18:0/1246975217 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa60c003520 con 0x7fa6200b4360 2024-03-29T11:09:06.410 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.422+0000 7fa61f7fe640 1 -- 172.21.15.18:0/1246975217 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 739668459 0 0) 0x7fa608001110 con 0x7fa62009ac40 2024-03-29T11:09:06.410 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.422+0000 7fa61f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:09:06.410 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.422+0000 7fa61f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:06.410 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.422+0000 7fa61f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:06.410 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.422+0000 7fa61f7fe640 10 monclient(hunting): my global_id is 4182 2024-03-29T11:09:06.410 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.422+0000 7fa61f7fe640 1 -- 172.21.15.18:0/1246975217 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6200b39c0 con 0x7fa62009ac40 2024-03-29T11:09:06.410 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.423+0000 7fa61f7fe640 1 -- 172.21.15.18:0/1246975217 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1597811391 0 0) 0x7fa62009da30 con 0x7fa62009c870 2024-03-29T11:09:06.410 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.423+0000 7fa61f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:09:06.410 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.423+0000 7fa61f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:06.410 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.423+0000 7fa61f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:06.410 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.423+0000 7fa61f7fe640 10 monclient(hunting): my global_id is 4180 2024-03-29T11:09:06.410 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.423+0000 7fa61f7fe640 1 -- 172.21.15.18:0/1246975217 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa608001110 con 0x7fa62009c870 2024-03-29T11:09:06.411 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.423+0000 7fa61f7fe640 1 -- 172.21.15.18:0/1246975217 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3719344363 0 0) 0x7fa60c003520 con 0x7fa6200b4360 2024-03-29T11:09:06.411 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.423+0000 7fa61f7fe640 1 -- 172.21.15.18:0/1246975217 >> v1:172.21.15.70:6789/0 conn(0x7fa62009c870 legacy=0x7fa62013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:06.411 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.423+0000 7fa61f7fe640 1 -- 172.21.15.18:0/1246975217 >> v1:172.21.15.18:6789/0 conn(0x7fa62009ac40 legacy=0x7fa6200b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:06.411 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.423+0000 7fa61f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:09:06.411 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.423+0000 7fa61f7fe640 1 monclient: found mon.noname-b 2024-03-29T11:09:06.411 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.423+0000 7fa61f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:06.411 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.423+0000 7fa61f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:09:06.411 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.423+0000 7fa61f7fe640 1 -- 172.21.15.18:0/1246975217 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6200b5160 con 0x7fa6200b4360 2024-03-29T11:09:06.411 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.423+0000 7fa61f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:09:06.411 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.423+0000 7fa61f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:09:06.411 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.423+0000 7fa61f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:06.411 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.423+0000 7fa61f7fe640 1 -- 172.21.15.18:0/1246975217 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fa618001a10 con 0x7fa6200b4360 2024-03-29T11:09:06.411 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.423+0000 7fa626f04640 5 monclient: authenticate success, global_id 4187 2024-03-29T11:09:06.411 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.423+0000 7fa626f04640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:06.411 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.423+0000 7fa61f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:06.412 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.423+0000 7fa61f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:09:06.412 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.423+0000 7fa61f7fe640 10 monclient: dump: 2024-03-29T11:09:06.412 INFO:teuthology.orchestra.run.smithi018.stderr:epoch 1 2024-03-29T11:09:06.412 INFO:teuthology.orchestra.run.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:06.412 INFO:teuthology.orchestra.run.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:06.412 INFO:teuthology.orchestra.run.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:06.412 INFO:teuthology.orchestra.run.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:06.412 INFO:teuthology.orchestra.run.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:06.412 INFO:teuthology.orchestra.run.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:06.412 INFO:teuthology.orchestra.run.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:06.412 INFO:teuthology.orchestra.run.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:06.412 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-29T11:09:06.412 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.423+0000 7fa626f04640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:06.412 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.423+0000 7fa61f7fe640 1 -- 172.21.15.18:0/1246975217 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa618001b70 con 0x7fa6200b4360 2024-03-29T11:09:06.412 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.423+0000 7fa61f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:06.413 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.423+0000 7fa61f7fe640 1 -- 172.21.15.18:0/1246975217 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fa618001fc0 con 0x7fa6200b4360 2024-03-29T11:09:06.413 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.423+0000 7fa626f04640 10 monclient: get_monmap_and_config success 2024-03-29T11:09:06.413 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.423+0000 7fa626f04640 4 set_mon_vals no callback set 2024-03-29T11:09:06.413 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.423+0000 7fa61f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:06.413 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.423+0000 7fa61f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:09:06.413 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.423+0000 7fa61f7fe640 10 monclient: dump: 2024-03-29T11:09:06.413 INFO:teuthology.orchestra.run.smithi018.stderr:epoch 1 2024-03-29T11:09:06.413 INFO:teuthology.orchestra.run.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:06.413 INFO:teuthology.orchestra.run.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:06.413 INFO:teuthology.orchestra.run.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:06.413 INFO:teuthology.orchestra.run.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:06.413 INFO:teuthology.orchestra.run.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:06.413 INFO:teuthology.orchestra.run.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:06.413 INFO:teuthology.orchestra.run.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:06.413 INFO:teuthology.orchestra.run.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:06.414 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-29T11:09:06.414 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.423+0000 7fa626f04640 10 monclient: shutdown 2024-03-29T11:09:06.414 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.423+0000 7fa626f04640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:06.414 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.423+0000 7fa626f04640 1 -- 172.21.15.18:0/1246975217 >> v1:172.21.15.18:6790/0 conn(0x7fa6200b4360 legacy=0x7fa62014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:06.414 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.423+0000 7fa626f04640 1 -- 172.21.15.18:0/1246975217 shutdown_connections 2024-03-29T11:09:06.414 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.423+0000 7fa626f04640 1 -- 172.21.15.18:0/1246975217 >> 172.21.15.18:0/1246975217 conn(0x7fa620066b00 msgr2=0x7fa620066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:06.414 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.423+0000 7fa626f04640 1 -- 172.21.15.18:0/1246975217 shutdown_connections 2024-03-29T11:09:06.414 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.423+0000 7fa626f04640 1 -- 172.21.15.18:0/1246975217 wait complete. 2024-03-29T11:09:06.414 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.424+0000 7fa626f04640 10 monclient: build_initial_monmap 2024-03-29T11:09:06.414 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.424+0000 7fa626f04640 10 monclient: monmap: 2024-03-29T11:09:06.414 INFO:teuthology.orchestra.run.smithi018.stderr:epoch 0 2024-03-29T11:09:06.414 INFO:teuthology.orchestra.run.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:09:06.414 INFO:teuthology.orchestra.run.smithi018.stderr:last_changed 0.000000 2024-03-29T11:09:06.414 INFO:teuthology.orchestra.run.smithi018.stderr:created 0.000000 2024-03-29T11:09:06.415 INFO:teuthology.orchestra.run.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:06.415 INFO:teuthology.orchestra.run.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:06.415 INFO:teuthology.orchestra.run.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:09:06.415 INFO:teuthology.orchestra.run.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:09:06.415 INFO:teuthology.orchestra.run.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:09:06.415 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-29T11:09:06.415 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.424+0000 7fa626f04640 1 librados: starting msgr at 2024-03-29T11:09:06.415 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.424+0000 7fa626f04640 1 librados: starting objecter 2024-03-29T11:09:06.415 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.424+0000 7fa626f04640 1 Processor -- start 2024-03-29T11:09:06.415 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.424+0000 7fa626f04640 1 -- start start 2024-03-29T11:09:06.415 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.424+0000 7fa626f04640 1 librados: setting wanted keys 2024-03-29T11:09:06.415 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.424+0000 7fa626f04640 1 librados: calling monclient init 2024-03-29T11:09:06.415 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.424+0000 7fa626f04640 10 monclient: init 2024-03-29T11:09:06.415 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.425+0000 7fa626f04640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:06.415 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.425+0000 7fa626f04640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:09:06.416 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.425+0000 7fa626f04640 10 monclient(hunting): picked mon.noname-c con 0x7fa62009ac40 addr v1:172.21.15.18:6790/0 2024-03-29T11:09:06.416 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.425+0000 7fa626f04640 10 monclient(hunting): picked mon.noname-a con 0x7fa620146060 addr v1:172.21.15.18:6789/0 2024-03-29T11:09:06.416 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.425+0000 7fa626f04640 10 monclient(hunting): picked mon.noname-b con 0x7fa620142110 addr v1:172.21.15.70:6789/0 2024-03-29T11:09:06.416 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.425+0000 7fa626f04640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6201397e0 con 0x7fa620146060 2024-03-29T11:09:06.416 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.425+0000 7fa626f04640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa620142cc0 con 0x7fa620142110 2024-03-29T11:09:06.416 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.425+0000 7fa626f04640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6201406b0 con 0x7fa62009ac40 2024-03-29T11:09:06.416 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.425+0000 7fa626f04640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:06.416 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.425+0000 7fa626f04640 10 monclient(hunting): authenticate will time out at 772.697576s 2024-03-29T11:09:06.416 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.425+0000 7fa61ffff640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fa620146060 0x7fa6201389c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:54964/0 (socket says 172.21.15.18:54964) 2024-03-29T11:09:06.416 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.425+0000 7fa624ca0640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fa62009ac40 0x7fa62013ffa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:51564/0 (socket says 172.21.15.18:51564) 2024-03-29T11:09:06.416 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.425+0000 7fa61ffff640 1 -- 172.21.15.18:0/116937480 learned_addr learned my addr 172.21.15.18:0/116937480 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:09:06.416 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.425+0000 7fa61d7fa640 10 client.?.objecter ms_handle_connect 0x7fa620146060 2024-03-29T11:09:06.416 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.425+0000 7fa61d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:06.416 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.425+0000 7fa61d7fa640 10 client.?.objecter ms_handle_connect 0x7fa62009ac40 2024-03-29T11:09:06.416 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.425+0000 7fa61d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:06.417 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.425+0000 7fa61d7fa640 10 client.?.objecter ms_handle_connect 0x7fa620142110 2024-03-29T11:09:06.417 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.425+0000 7fa61d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:06.417 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 1 -- 172.21.15.18:0/116937480 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1178645268 0 0) 0x7fa6201406b0 con 0x7fa62009ac40 2024-03-29T11:09:06.417 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:09:06.417 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:06.417 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:06.417 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 10 monclient(hunting): my global_id is 4190 2024-03-29T11:09:06.417 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 1 -- 172.21.15.18:0/116937480 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa600004eb0 con 0x7fa62009ac40 2024-03-29T11:09:06.417 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 1 -- 172.21.15.18:0/116937480 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1581093531 0 0) 0x7fa6201397e0 con 0x7fa620146060 2024-03-29T11:09:06.417 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:09:06.417 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:06.417 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:06.417 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 10 monclient(hunting): my global_id is 4185 2024-03-29T11:09:06.417 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 1 -- 172.21.15.18:0/116937480 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6201406b0 con 0x7fa620146060 2024-03-29T11:09:06.417 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 1 -- 172.21.15.18:0/116937480 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3387373626 0 0) 0x7fa620142cc0 con 0x7fa620142110 2024-03-29T11:09:06.418 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:09:06.418 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:06.418 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:06.418 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 10 monclient(hunting): my global_id is 4183 2024-03-29T11:09:06.418 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 1 -- 172.21.15.18:0/116937480 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6201397e0 con 0x7fa620142110 2024-03-29T11:09:06.418 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 1 -- 172.21.15.18:0/116937480 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 188835421 0 0) 0x7fa600004eb0 con 0x7fa62009ac40 2024-03-29T11:09:06.418 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 1 -- 172.21.15.18:0/116937480 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa620142cc0 con 0x7fa62009ac40 2024-03-29T11:09:06.418 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 1 -- 172.21.15.18:0/116937480 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fa608001ac0 con 0x7fa62009ac40 2024-03-29T11:09:06.418 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:06.418 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:09:06.418 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 10 monclient(hunting): dump: 2024-03-29T11:09:06.418 INFO:teuthology.orchestra.run.smithi018.stderr:epoch 1 2024-03-29T11:09:06.418 INFO:teuthology.orchestra.run.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:06.418 INFO:teuthology.orchestra.run.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:06.418 INFO:teuthology.orchestra.run.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:06.419 INFO:teuthology.orchestra.run.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:06.419 INFO:teuthology.orchestra.run.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:06.419 INFO:teuthology.orchestra.run.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:06.419 INFO:teuthology.orchestra.run.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:06.419 INFO:teuthology.orchestra.run.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:06.419 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-29T11:09:06.419 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:09:06.419 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 1 -- 172.21.15.18:0/116937480 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3362458061 0 0) 0x7fa6201397e0 con 0x7fa620142110 2024-03-29T11:09:06.419 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 1 -- 172.21.15.18:0/116937480 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa600004eb0 con 0x7fa620142110 2024-03-29T11:09:06.419 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 1 -- 172.21.15.18:0/116937480 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 992775025 0 0) 0x7fa6201406b0 con 0x7fa620146060 2024-03-29T11:09:06.419 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 1 -- 172.21.15.18:0/116937480 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6201397e0 con 0x7fa620146060 2024-03-29T11:09:06.419 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 1 -- 172.21.15.18:0/116937480 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fa618002100 con 0x7fa620142110 2024-03-29T11:09:06.419 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:06.419 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:09:06.419 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 10 monclient(hunting): dump: 2024-03-29T11:09:06.420 INFO:teuthology.orchestra.run.smithi018.stderr:epoch 1 2024-03-29T11:09:06.420 INFO:teuthology.orchestra.run.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:06.420 INFO:teuthology.orchestra.run.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:06.420 INFO:teuthology.orchestra.run.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:06.420 INFO:teuthology.orchestra.run.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:06.420 INFO:teuthology.orchestra.run.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:06.420 INFO:teuthology.orchestra.run.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:06.420 INFO:teuthology.orchestra.run.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:06.420 INFO:teuthology.orchestra.run.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:06.420 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-29T11:09:06.420 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 1 -- 172.21.15.18:0/116937480 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fa6100018f0 con 0x7fa620146060 2024-03-29T11:09:06.420 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:06.420 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:09:06.420 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 10 monclient(hunting): dump: 2024-03-29T11:09:06.420 INFO:teuthology.orchestra.run.smithi018.stderr:epoch 1 2024-03-29T11:09:06.421 INFO:teuthology.orchestra.run.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:06.421 INFO:teuthology.orchestra.run.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:06.421 INFO:teuthology.orchestra.run.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:06.421 INFO:teuthology.orchestra.run.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:06.421 INFO:teuthology.orchestra.run.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:06.421 INFO:teuthology.orchestra.run.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:06.421 INFO:teuthology.orchestra.run.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:06.421 INFO:teuthology.orchestra.run.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:06.421 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-29T11:09:06.421 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 1 -- 172.21.15.18:0/116937480 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 384533421 0 0) 0x7fa620142cc0 con 0x7fa62009ac40 2024-03-29T11:09:06.421 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 1 -- 172.21.15.18:0/116937480 >> v1:172.21.15.70:6789/0 conn(0x7fa620142110 legacy=0x7fa6201390d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:06.421 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 1 -- 172.21.15.18:0/116937480 >> v1:172.21.15.18:6789/0 conn(0x7fa620146060 legacy=0x7fa6201389c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:06.421 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:09:06.421 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 1 monclient: found mon.c 2024-03-29T11:09:06.421 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:06.422 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:09:06.422 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 1 -- 172.21.15.18:0/116937480 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa620140840 con 0x7fa62009ac40 2024-03-29T11:09:06.422 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:09:06.422 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:09:06.422 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:06.422 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa626f04640 5 monclient: authenticate success, global_id 4190 2024-03-29T11:09:06.422 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa626f04640 10 monclient: _renew_subs 2024-03-29T11:09:06.422 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa626f04640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:09:06.422 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa626f04640 1 -- 172.21.15.18:0/116937480 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa620140a90 con 0x7fa62009ac40 2024-03-29T11:09:06.422 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa626f04640 10 client.4190.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:09:06.422 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa626f04640 10 monclient: _renew_subs 2024-03-29T11:09:06.422 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa626f04640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:09:06.422 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa626f04640 1 -- 172.21.15.18:0/116937480 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6200acee0 con 0x7fa62009ac40 2024-03-29T11:09:06.422 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa626f04640 1 librados: init done 2024-03-29T11:09:06.422 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 1 -- 172.21.15.18:0/116937480 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa608001d20 con 0x7fa62009ac40 2024-03-29T11:09:06.423 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:06.423 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 1 -- 172.21.15.18:0/116937480 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fa608004c30 con 0x7fa62009ac40 2024-03-29T11:09:06.423 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:06.423 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:09:06.423 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 10 monclient: dump: 2024-03-29T11:09:06.423 INFO:teuthology.orchestra.run.smithi018.stderr:epoch 1 2024-03-29T11:09:06.423 INFO:teuthology.orchestra.run.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:06.423 INFO:teuthology.orchestra.run.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:06.423 INFO:teuthology.orchestra.run.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:06.423 INFO:teuthology.orchestra.run.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:06.423 INFO:teuthology.orchestra.run.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:06.423 INFO:teuthology.orchestra.run.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:06.423 INFO:teuthology.orchestra.run.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:06.423 INFO:teuthology.orchestra.run.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:06.423 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-29T11:09:06.424 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61effd640 4 set_mon_vals no callback set 2024-03-29T11:09:06.424 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 1 -- 172.21.15.18:0/116937480 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fa6080051e0 con 0x7fa62009ac40 2024-03-29T11:09:06.424 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 1 -- 172.21.15.18:0/116937480 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7fa608006af0 con 0x7fa62009ac40 2024-03-29T11:09:06.424 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 10 client.4190.objecter ms_dispatch 0x7fa620000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:09:06.424 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 3 client.4190.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:09:06.424 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 3 client.4190.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:09:06.424 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.426+0000 7fa61d7fa640 20 client.4190.objecter dump_active .. 0 homeless 2024-03-29T11:09:06.424 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.427+0000 7fa626f04640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:06.424 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.427+0000 7fa626f04640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:06.424 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.427+0000 7fa626f04640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:09:06.424 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.427+0000 7fa626f04640 1 -- 172.21.15.18:0/116937480 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa604005080 con 0x7fa62009ac40 2024-03-29T11:09:06.424 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.429+0000 7fa61d7fa640 1 -- 172.21.15.18:0/116937480 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fa608006420 con 0x7fa62009ac40 2024-03-29T11:09:06.424 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.429+0000 7fa61d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:06.424 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.429+0000 7fa61d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:09:06.450 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-29T11:09:06.472+0000 7fad3de2e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:09:06.469 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.491+0000 7fa626f04640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:06.469 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.491+0000 7fa626f04640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:06.469 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.491+0000 7fa626f04640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:09:06.470 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.491+0000 7fa626f04640 1 -- 172.21.15.18:0/116937480 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa604005330 con 0x7fa62009ac40 2024-03-29T11:09:06.470 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.492+0000 7fa61d7fa640 1 -- 172.21.15.18:0/116937480 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7fa608006790 con 0x7fa62009ac40 2024-03-29T11:09:06.470 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.492+0000 7fa61d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:06.470 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.492+0000 7fa61d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:09:06.470 INFO:teuthology.orchestra.run.smithi018.stdout: 2024-03-29T11:09:06.471 INFO:teuthology.orchestra.run.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:09:06.472 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.492+0000 7fa626f04640 10 librados: watch_flush enter 2024-03-29T11:09:06.472 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.492+0000 7fa626f04640 10 librados: watch_flush exit 2024-03-29T11:09:06.472 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.492+0000 7fa626f04640 20 client.4190.objecter shutdown clearing up homeless session... 2024-03-29T11:09:06.472 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.492+0000 7fa626f04640 10 client.4190.objecter successfully canceled tick 2024-03-29T11:09:06.472 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.492+0000 7fa626f04640 10 monclient: shutdown 2024-03-29T11:09:06.472 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.492+0000 7fa626f04640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:06.472 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.492+0000 7fa626f04640 1 -- 172.21.15.18:0/116937480 >> v1:172.21.15.18:6790/0 conn(0x7fa62009ac40 legacy=0x7fa62013ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:06.472 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.493+0000 7fa626f04640 1 -- 172.21.15.18:0/116937480 shutdown_connections 2024-03-29T11:09:06.472 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.493+0000 7fa626f04640 1 -- 172.21.15.18:0/116937480 >> 172.21.15.18:0/116937480 conn(0x7fa620066b00 msgr2=0x7fa62013ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:06.472 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.493+0000 7fa626f04640 1 -- 172.21.15.18:0/116937480 shutdown_connections 2024-03-29T11:09:06.472 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.493+0000 7fa626f04640 1 -- 172.21.15.18:0/116937480 wait complete. 2024-03-29T11:09:06.472 INFO:teuthology.orchestra.run.smithi018.stderr:2024-03-29T11:09:06.493+0000 7fa626f04640 1 librados: shutdown 2024-03-29T11:09:06.474 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-03-29T11:09:06.495+0000 7f6963a2c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:09:06.481 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T11:09:06.481 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T11:09:06.499 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-29T11:09:06.521+0000 7fad3de2e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:09:06.518 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-03-29T11:09:06.539+0000 7f6963a2c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:09:06.617 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-29T11:09:06.638+0000 7fad3de2e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:09:06.630 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-03-29T11:09:06.651+0000 7f6963a2c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:09:06.716 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-29T11:09:06.737+0000 7fad3de2e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:09:06.723 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-03-29T11:09:06.744+0000 7f6963a2c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:09:06.847 INFO:tasks.ceph.osd.0.smithi018.stderr:2024-03-29T11:09:06.868+0000 7f67224bd640 -1 Falling back to public interface 2024-03-29T11:09:06.850 INFO:tasks.ceph.osd.2.smithi018.stderr:2024-03-29T11:09:06.872+0000 7ff1e2549640 -1 Falling back to public interface 2024-03-29T11:09:06.854 INFO:tasks.ceph.osd.4.smithi070.stderr:2024-03-29T11:09:06.875+0000 7fc94da04640 -1 Falling back to public interface 2024-03-29T11:09:06.855 INFO:tasks.ceph.osd.1.smithi018.stderr:2024-03-29T11:09:06.877+0000 7f036e415640 -1 Falling back to public interface 2024-03-29T11:09:06.859 INFO:tasks.ceph.osd.3.smithi018.stderr:2024-03-29T11:09:06.880+0000 7f712fe0f640 -1 Falling back to public interface 2024-03-29T11:09:06.860 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-03-29T11:09:06.881+0000 7f5cc7b2a640 -1 Falling back to public interface 2024-03-29T11:09:06.867 INFO:tasks.ceph.osd.5.smithi070.stderr:2024-03-29T11:09:06.889+0000 7f4022619640 -1 Falling back to public interface 2024-03-29T11:09:06.872 INFO:tasks.ceph.osd.7.smithi070.stderr:2024-03-29T11:09:06.894+0000 7fedc00d3640 -1 Falling back to public interface 2024-03-29T11:09:06.878 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-29T11:09:06.899+0000 7fad3de2e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:09:06.879 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-03-29T11:09:06.901+0000 7f6963a2c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:09:06.922 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-29T11:09:06.943+0000 7fad3de2e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:09:06.922 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-03-29T11:09:06.944+0000 7f6963a2c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:09:07.073 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-03-29T11:09:07.095+0000 7f6963a2c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:09:07.083 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-29T11:09:07.105+0000 7fad3de2e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:09:07.286 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:09:07.375 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.397+0000 7f6da1b30640 10 monclient: get_monmap_and_config 2024-03-29T11:09:07.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.397+0000 7f6da1b30640 10 monclient: build_initial_monmap 2024-03-29T11:09:07.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.397+0000 7f6da1b30640 10 monclient: monmap: 2024-03-29T11:09:07.376 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:09:07.376 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:07.376 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:09:07.398550+0000 2024-03-29T11:09:07.376 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:09:07.398550+0000 2024-03-29T11:09:07.376 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:07.376 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:07.376 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:09:07.376 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:09:07.376 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:09:07.376 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:07.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.398+0000 7f6da1b30640 1 Processor -- start 2024-03-29T11:09:07.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.398+0000 7f6da1b30640 1 -- start start 2024-03-29T11:09:07.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.398+0000 7f6da1b30640 10 monclient: init 2024-03-29T11:09:07.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.398+0000 7f6da1b30640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:07.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.398+0000 7f6da1b30640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:09:07.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.398+0000 7f6da1b30640 10 monclient(hunting): picked mon.noname-c con 0x7f6d9c1422b0 addr v1:172.21.15.70:6789/0 2024-03-29T11:09:07.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.398+0000 7f6da1b30640 10 monclient(hunting): picked mon.noname-a con 0x7f6d9c142690 addr v1:172.21.15.18:6789/0 2024-03-29T11:09:07.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.398+0000 7f6da1b30640 10 monclient(hunting): picked mon.noname-b con 0x7f6d9c0b17d0 addr v1:172.21.15.18:6790/0 2024-03-29T11:09:07.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.398+0000 7f6da1b30640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9c142c80 con 0x7f6d9c142690 2024-03-29T11:09:07.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.398+0000 7f6da1b30640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9c0aae40 con 0x7f6d9c1422b0 2024-03-29T11:09:07.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6da1b30640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9c13fbd0 con 0x7f6d9c0b17d0 2024-03-29T11:09:07.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6da1b30640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:07.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6da1b30640 10 monclient(hunting): authenticate will time out at 773.670930s 2024-03-29T11:09:07.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9affd640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f6d9c142690 0x7f6d9c0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:34796/0 (socket says 172.21.15.18:34796) 2024-03-29T11:09:07.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9bfff640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f6d9c0b17d0 0x7f6d9c0b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:60792/0 (socket says 172.21.15.18:60792) 2024-03-29T11:09:07.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9affd640 1 -- 172.21.15.18:0/58574739 learned_addr learned my addr 172.21.15.18:0/58574739 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:09:07.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9a7fc640 1 -- 172.21.15.18:0/58574739 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 819228449 0 0) 0x7f6d9c142c80 con 0x7f6d9c142690 2024-03-29T11:09:07.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:09:07.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:07.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:07.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9a7fc640 10 monclient(hunting): my global_id is 4188 2024-03-29T11:09:07.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9a7fc640 1 -- 172.21.15.18:0/58574739 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d88003520 con 0x7f6d9c142690 2024-03-29T11:09:07.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9a7fc640 1 -- 172.21.15.18:0/58574739 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 637168346 0 0) 0x7f6d9c13fbd0 con 0x7f6d9c0b17d0 2024-03-29T11:09:07.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:09:07.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:07.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:07.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9a7fc640 10 monclient(hunting): my global_id is 4193 2024-03-29T11:09:07.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9a7fc640 1 -- 172.21.15.18:0/58574739 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d9c142c80 con 0x7f6d9c0b17d0 2024-03-29T11:09:07.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9a7fc640 1 -- 172.21.15.18:0/58574739 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2533798023 0 0) 0x7f6d9c0aae40 con 0x7f6d9c1422b0 2024-03-29T11:09:07.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:09:07.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:07.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:07.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9a7fc640 10 monclient(hunting): my global_id is 4186 2024-03-29T11:09:07.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9a7fc640 1 -- 172.21.15.18:0/58574739 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d9c13fbd0 con 0x7f6d9c1422b0 2024-03-29T11:09:07.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9a7fc640 1 -- 172.21.15.18:0/58574739 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2754021966 0 0) 0x7f6d88003520 con 0x7f6d9c142690 2024-03-29T11:09:07.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9a7fc640 1 -- 172.21.15.18:0/58574739 >> v1:172.21.15.18:6790/0 conn(0x7f6d9c0b17d0 legacy=0x7f6d9c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:07.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9a7fc640 1 -- 172.21.15.18:0/58574739 >> v1:172.21.15.70:6789/0 conn(0x7f6d9c1422b0 legacy=0x7f6d9c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:07.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:09:07.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9a7fc640 1 monclient: found mon.noname-a 2024-03-29T11:09:07.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:07.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-29T11:09:07.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9a7fc640 1 -- 172.21.15.18:0/58574739 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d9c151940 con 0x7f6d9c142690 2024-03-29T11:09:07.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:09:07.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:09:07.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:07.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9a7fc640 1 -- 172.21.15.18:0/58574739 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f6d84001690 con 0x7f6d9c142690 2024-03-29T11:09:07.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:07.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:09:07.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9a7fc640 10 monclient: dump: 2024-03-29T11:09:07.389 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:09:07.389 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:07.389 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:07.389 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:07.389 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:07.389 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:07.389 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:07.389 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:07.389 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:07.389 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:07.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6da1b30640 5 monclient: authenticate success, global_id 4188 2024-03-29T11:09:07.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6da1b30640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:07.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9a7fc640 1 -- 172.21.15.18:0/58574739 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d84001870 con 0x7f6d9c142690 2024-03-29T11:09:07.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:07.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9a7fc640 1 -- 172.21.15.18:0/58574739 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f6d84001c40 con 0x7f6d9c142690 2024-03-29T11:09:07.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:07.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:09:07.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6d9a7fc640 10 monclient: dump: 2024-03-29T11:09:07.390 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:09:07.390 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:07.390 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:07.390 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:07.390 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:07.390 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:07.390 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:07.390 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:07.390 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:07.390 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:07.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6da1b30640 10 monclient: get_monmap_and_config success 2024-03-29T11:09:07.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.399+0000 7f6da1b30640 4 set_mon_vals no callback set 2024-03-29T11:09:07.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.400+0000 7f6da1b30640 10 monclient: shutdown 2024-03-29T11:09:07.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.400+0000 7f6da1b30640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:07.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.400+0000 7f6da1b30640 1 -- 172.21.15.18:0/58574739 >> v1:172.21.15.18:6789/0 conn(0x7f6d9c142690 legacy=0x7f6d9c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:07.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.400+0000 7f6da1b30640 1 -- 172.21.15.18:0/58574739 shutdown_connections 2024-03-29T11:09:07.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.400+0000 7f6da1b30640 1 -- 172.21.15.18:0/58574739 >> 172.21.15.18:0/58574739 conn(0x7f6d9c066a00 msgr2=0x7f6d9c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:07.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.400+0000 7f6da1b30640 1 -- 172.21.15.18:0/58574739 shutdown_connections 2024-03-29T11:09:07.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.400+0000 7f6da1b30640 1 -- 172.21.15.18:0/58574739 wait complete. 2024-03-29T11:09:07.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.401+0000 7f6da1b30640 10 monclient: build_initial_monmap 2024-03-29T11:09:07.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.401+0000 7f6da1b30640 10 monclient: monmap: 2024-03-29T11:09:07.391 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:09:07.391 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:09:07.391 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:09:07.392 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:09:07.392 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:07.392 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:07.392 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:09:07.392 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:09:07.392 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:09:07.392 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:07.392 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.401+0000 7f6da1b30640 1 librados: starting msgr at 2024-03-29T11:09:07.392 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.401+0000 7f6da1b30640 1 librados: starting objecter 2024-03-29T11:09:07.392 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.401+0000 7f6da1b30640 1 Processor -- start 2024-03-29T11:09:07.392 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.401+0000 7f6da1b30640 1 -- start start 2024-03-29T11:09:07.392 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.401+0000 7f6da1b30640 1 librados: setting wanted keys 2024-03-29T11:09:07.392 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.401+0000 7f6da1b30640 1 librados: calling monclient init 2024-03-29T11:09:07.392 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.401+0000 7f6da1b30640 10 monclient: init 2024-03-29T11:09:07.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6da1b30640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:07.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6da1b30640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:09:07.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6da1b30640 10 monclient(hunting): picked mon.noname-a con 0x7f6d9c0b17d0 addr v1:172.21.15.18:6789/0 2024-03-29T11:09:07.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6da1b30640 10 monclient(hunting): picked mon.noname-b con 0x7f6d9c1422b0 addr v1:172.21.15.70:6789/0 2024-03-29T11:09:07.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6da1b30640 10 monclient(hunting): picked mon.noname-c con 0x7f6d9c142690 addr v1:172.21.15.18:6790/0 2024-03-29T11:09:07.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6da1b30640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9c0aced0 con 0x7f6d9c0b17d0 2024-03-29T11:09:07.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6da1b30640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9c0ad120 con 0x7f6d9c1422b0 2024-03-29T11:09:07.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6da1b30640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d9c132600 con 0x7f6d9c142690 2024-03-29T11:09:07.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6da1b30640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:07.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6da1b30640 10 monclient(hunting): authenticate will time out at 773.674256s 2024-03-29T11:09:07.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6d9bfff640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f6d9c142690 0x7f6d9c130cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:60798/0 (socket says 172.21.15.18:60798) 2024-03-29T11:09:07.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6d9bfff640 1 -- 172.21.15.18:0/3772180379 learned_addr learned my addr 172.21.15.18:0/3772180379 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:09:07.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6d9b7fe640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f6d9c0b17d0 0x7f6d9c0ab7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:34812/0 (socket says 172.21.15.18:34812) 2024-03-29T11:09:07.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6d93fff640 10 client.?.objecter ms_handle_connect 0x7f6d9c0b17d0 2024-03-29T11:09:07.394 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6d93fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:07.394 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6d93fff640 10 client.?.objecter ms_handle_connect 0x7f6d9c142690 2024-03-29T11:09:07.394 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6d93fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:07.394 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6d93fff640 10 client.?.objecter ms_handle_connect 0x7f6d9c1422b0 2024-03-29T11:09:07.394 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6d93fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:07.394 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6d93fff640 1 -- 172.21.15.18:0/3772180379 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2812505112 0 0) 0x7f6d9c0aced0 con 0x7f6d9c0b17d0 2024-03-29T11:09:07.394 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6d93fff640 10 monclient(hunting): _negotiate 2024-03-29T11:09:07.394 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6d93fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:07.394 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6d93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:07.394 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6d93fff640 10 monclient(hunting): my global_id is 4191 2024-03-29T11:09:07.394 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6d93fff640 1 -- 172.21.15.18:0/3772180379 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d7c004eb0 con 0x7f6d9c0b17d0 2024-03-29T11:09:07.394 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6d93fff640 1 -- 172.21.15.18:0/3772180379 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 630968287 0 0) 0x7f6d9c132600 con 0x7f6d9c142690 2024-03-29T11:09:07.394 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6d93fff640 10 monclient(hunting): _negotiate 2024-03-29T11:09:07.394 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6d93fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:07.395 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6d93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:07.395 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6d93fff640 10 monclient(hunting): my global_id is 4196 2024-03-29T11:09:07.395 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6d93fff640 1 -- 172.21.15.18:0/3772180379 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d9c0aced0 con 0x7f6d9c142690 2024-03-29T11:09:07.395 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6d93fff640 1 -- 172.21.15.18:0/3772180379 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2886150913 0 0) 0x7f6d9c0ad120 con 0x7f6d9c1422b0 2024-03-29T11:09:07.395 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6d93fff640 10 monclient(hunting): _negotiate 2024-03-29T11:09:07.395 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6d93fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:07.395 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6d93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:07.395 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6d93fff640 10 monclient(hunting): my global_id is 4189 2024-03-29T11:09:07.395 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6d93fff640 1 -- 172.21.15.18:0/3772180379 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d9c132600 con 0x7f6d9c1422b0 2024-03-29T11:09:07.395 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6d93fff640 1 -- 172.21.15.18:0/3772180379 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1822524679 0 0) 0x7f6d7c004eb0 con 0x7f6d9c0b17d0 2024-03-29T11:09:07.395 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6d93fff640 1 -- 172.21.15.18:0/3772180379 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d9c0ad120 con 0x7f6d9c0b17d0 2024-03-29T11:09:07.395 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6d93fff640 1 -- 172.21.15.18:0/3772180379 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f6d8c001970 con 0x7f6d9c0b17d0 2024-03-29T11:09:07.395 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6d93fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:07.395 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6d93fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:09:07.395 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6d93fff640 10 monclient(hunting): dump: 2024-03-29T11:09:07.396 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:09:07.396 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:07.396 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:07.396 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:07.396 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:07.396 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:07.396 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:07.396 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:07.396 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:07.396 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:07.396 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6d93fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:09:07.396 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.402+0000 7f6d93fff640 1 -- 172.21.15.18:0/3772180379 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1217553683 0 0) 0x7f6d9c0aced0 con 0x7f6d9c142690 2024-03-29T11:09:07.396 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6d93fff640 1 -- 172.21.15.18:0/3772180379 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d7c004eb0 con 0x7f6d9c142690 2024-03-29T11:09:07.396 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6d93fff640 1 -- 172.21.15.18:0/3772180379 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f6d94001610 con 0x7f6d9c142690 2024-03-29T11:09:07.396 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6d93fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:07.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6d93fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:09:07.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6d93fff640 10 monclient(hunting): dump: 2024-03-29T11:09:07.397 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:09:07.397 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:07.397 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:07.397 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:07.397 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:07.397 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:07.397 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:07.397 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:07.397 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:07.397 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:07.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6d93fff640 1 -- 172.21.15.18:0/3772180379 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 486651522 0 0) 0x7f6d9c132600 con 0x7f6d9c1422b0 2024-03-29T11:09:07.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6d93fff640 1 -- 172.21.15.18:0/3772180379 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d9c0aced0 con 0x7f6d9c1422b0 2024-03-29T11:09:07.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6d93fff640 1 -- 172.21.15.18:0/3772180379 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f6d84001b80 con 0x7f6d9c1422b0 2024-03-29T11:09:07.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6d93fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:07.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6d93fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:09:07.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6d93fff640 10 monclient(hunting): dump: 2024-03-29T11:09:07.398 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:09:07.398 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:07.398 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:07.398 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:07.398 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:07.398 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:07.398 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:07.398 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:07.398 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:07.398 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:07.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6d93fff640 1 -- 172.21.15.18:0/3772180379 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3383575077 0 0) 0x7f6d9c0ad120 con 0x7f6d9c0b17d0 2024-03-29T11:09:07.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6d93fff640 1 -- 172.21.15.18:0/3772180379 >> v1:172.21.15.18:6790/0 conn(0x7f6d9c142690 legacy=0x7f6d9c130cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:07.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6d93fff640 1 -- 172.21.15.18:0/3772180379 >> v1:172.21.15.70:6789/0 conn(0x7f6d9c1422b0 legacy=0x7f6d9c1305b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:07.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6d93fff640 10 monclient: _finish_hunting 0 2024-03-29T11:09:07.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6d93fff640 1 monclient: found mon.a 2024-03-29T11:09:07.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6d93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:07.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6d93fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:09:07.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6d93fff640 1 -- 172.21.15.18:0/3772180379 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d9c132a50 con 0x7f6d9c0b17d0 2024-03-29T11:09:07.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6d93fff640 10 monclient: _finish_auth 0 2024-03-29T11:09:07.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6d93fff640 10 monclient: _check_auth_tickets 2024-03-29T11:09:07.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6d93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:07.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6da1b30640 5 monclient: authenticate success, global_id 4191 2024-03-29T11:09:07.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6da1b30640 10 monclient: _renew_subs 2024-03-29T11:09:07.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6da1b30640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:09:07.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6da1b30640 1 -- 172.21.15.18:0/3772180379 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d9c1e2f50 con 0x7f6d9c0b17d0 2024-03-29T11:09:07.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6da1b30640 10 client.4191.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:09:07.400 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6da1b30640 10 monclient: _renew_subs 2024-03-29T11:09:07.400 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6da1b30640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:09:07.400 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6da1b30640 1 -- 172.21.15.18:0/3772180379 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d9c1e32c0 con 0x7f6d9c0b17d0 2024-03-29T11:09:07.400 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6da1b30640 1 librados: init done 2024-03-29T11:09:07.400 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6d93fff640 1 -- 172.21.15.18:0/3772180379 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d8c003610 con 0x7f6d9c0b17d0 2024-03-29T11:09:07.400 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6d93fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:07.400 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6d93fff640 1 -- 172.21.15.18:0/3772180379 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f6d8c003960 con 0x7f6d9c0b17d0 2024-03-29T11:09:07.400 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6d93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:07.400 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6d93fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:09:07.400 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6d99ffb640 4 set_mon_vals no callback set 2024-03-29T11:09:07.400 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6d93fff640 10 monclient: dump: 2024-03-29T11:09:07.400 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:09:07.400 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:07.400 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:07.400 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:07.401 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:07.401 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:07.401 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:07.401 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:07.401 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:07.401 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:07.401 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6d93fff640 1 -- 172.21.15.18:0/3772180379 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6d8c003ed0 con 0x7f6d9c0b17d0 2024-03-29T11:09:07.401 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6d93fff640 1 -- 172.21.15.18:0/3772180379 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f6d8c006970 con 0x7f6d9c0b17d0 2024-03-29T11:09:07.401 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6d93fff640 10 client.4191.objecter ms_dispatch 0x7f6d9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:09:07.401 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6d93fff640 3 client.4191.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:09:07.401 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6d93fff640 3 client.4191.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:09:07.401 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.403+0000 7f6d93fff640 20 client.4191.objecter dump_active .. 0 homeless 2024-03-29T11:09:07.401 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.404+0000 7f6da1b30640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:07.401 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.404+0000 7f6da1b30640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:07.401 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.404+0000 7f6da1b30640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:09:07.402 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.404+0000 7f6da1b30640 1 -- 172.21.15.18:0/3772180379 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d80005080 con 0x7f6d9c0b17d0 2024-03-29T11:09:07.402 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.406+0000 7f6d93fff640 1 -- 172.21.15.18:0/3772180379 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6d8c004360 con 0x7f6d9c0b17d0 2024-03-29T11:09:07.402 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.406+0000 7f6d93fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:07.402 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.406+0000 7f6d93fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:09:07.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.469+0000 7f6da1b30640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:07.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.469+0000 7f6da1b30640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:07.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.469+0000 7f6da1b30640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:09:07.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.469+0000 7f6da1b30640 1 -- 172.21.15.18:0/3772180379 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d80005330 con 0x7f6d9c0b17d0 2024-03-29T11:09:07.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.469+0000 7f6d93fff640 1 -- 172.21.15.18:0/3772180379 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f6d8c0063b0 con 0x7f6d9c0b17d0 2024-03-29T11:09:07.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.469+0000 7f6d93fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:07.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.469+0000 7f6d93fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:09:07.450 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:09:07.450 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:09:07.451 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.470+0000 7f6da1b30640 10 librados: watch_flush enter 2024-03-29T11:09:07.451 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.470+0000 7f6da1b30640 10 librados: watch_flush exit 2024-03-29T11:09:07.451 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.470+0000 7f6da1b30640 20 client.4191.objecter shutdown clearing up homeless session... 2024-03-29T11:09:07.451 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.470+0000 7f6da1b30640 10 client.4191.objecter successfully canceled tick 2024-03-29T11:09:07.451 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.470+0000 7f6da1b30640 10 monclient: shutdown 2024-03-29T11:09:07.451 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.470+0000 7f6da1b30640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:07.451 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.470+0000 7f6da1b30640 1 -- 172.21.15.18:0/3772180379 >> v1:172.21.15.18:6789/0 conn(0x7f6d9c0b17d0 legacy=0x7f6d9c0ab7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:07.451 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.470+0000 7f6da1b30640 1 -- 172.21.15.18:0/3772180379 shutdown_connections 2024-03-29T11:09:07.451 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.470+0000 7f6da1b30640 1 -- 172.21.15.18:0/3772180379 >> 172.21.15.18:0/3772180379 conn(0x7f6d9c066a00 msgr2=0x7f6d9c0b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:07.452 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.470+0000 7f6da1b30640 1 -- 172.21.15.18:0/3772180379 shutdown_connections 2024-03-29T11:09:07.452 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.470+0000 7f6da1b30640 1 -- 172.21.15.18:0/3772180379 wait complete. 2024-03-29T11:09:07.452 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:07.470+0000 7f6da1b30640 1 librados: shutdown 2024-03-29T11:09:07.458 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:09:07.887 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-03-29T11:09:07.908+0000 7f6963a2c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:09:07.920 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-29T11:09:07.942+0000 7fad3de2e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:09:08.158 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-03-29T11:09:08.179+0000 7f6963a2c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:09:08.206 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-29T11:09:08.228+0000 7fad3de2e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:09:08.208 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-03-29T11:09:08.230+0000 7f6963a2c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:09:08.259 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-29T11:09:08.281+0000 7fad3de2e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:09:08.323 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-03-29T11:09:08.345+0000 7f6963a2c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:09:08.385 INFO:tasks.ceph.mgr.y.smithi018.stderr:2024-03-29T11:09:08.407+0000 7fad3de2e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:09:09.338 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-03-29T11:09:09.359+0000 7f6903f5a640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T11:09:09.339 INFO:tasks.ceph.mgr.x.smithi070.stderr:2024-03-29T11:09:09.360+0000 7f6903f5a640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T11:09:10.340 INFO:tasks.ceph.osd.4.smithi070.stderr:2024-03-29T11:09:10.362+0000 7fc94da04640 -1 osd.4 0 log_to_monitors true 2024-03-29T11:09:10.376 INFO:tasks.ceph.osd.0.smithi018.stderr:2024-03-29T11:09:10.397+0000 7f67224bd640 -1 osd.0 0 log_to_monitors true 2024-03-29T11:09:10.381 INFO:tasks.ceph.osd.1.smithi018.stderr:2024-03-29T11:09:10.403+0000 7f036e415640 -1 osd.1 0 log_to_monitors true 2024-03-29T11:09:10.384 INFO:tasks.ceph.osd.2.smithi018.stderr:2024-03-29T11:09:10.405+0000 7ff1e2549640 -1 osd.2 0 log_to_monitors true 2024-03-29T11:09:10.390 INFO:tasks.ceph.osd.7.smithi070.stderr:2024-03-29T11:09:10.411+0000 7fedc00d3640 -1 osd.7 0 log_to_monitors true 2024-03-29T11:09:10.393 INFO:tasks.ceph.osd.6.smithi070.stderr:2024-03-29T11:09:10.415+0000 7f5cc7b2a640 -1 osd.6 0 log_to_monitors true 2024-03-29T11:09:10.409 INFO:tasks.ceph.osd.5.smithi070.stderr:2024-03-29T11:09:10.431+0000 7f4022619640 -1 osd.5 0 log_to_monitors true 2024-03-29T11:09:10.410 INFO:tasks.ceph.osd.3.smithi018.stderr:2024-03-29T11:09:10.432+0000 7f712fe0f640 -1 osd.3 0 log_to_monitors true 2024-03-29T11:09:14.264 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:09:14.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.441+0000 7f70152d7640 10 monclient: get_monmap_and_config 2024-03-29T11:09:14.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.441+0000 7f70152d7640 10 monclient: build_initial_monmap 2024-03-29T11:09:14.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.441+0000 7f70152d7640 10 monclient: monmap: 2024-03-29T11:09:14.420 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:09:14.420 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:14.420 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:09:14.442738+0000 2024-03-29T11:09:14.421 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:09:14.442738+0000 2024-03-29T11:09:14.421 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:14.421 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:14.421 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:09:14.421 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:09:14.421 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:09:14.421 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:14.422 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.443+0000 7f70152d7640 1 Processor -- start 2024-03-29T11:09:14.422 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.443+0000 7f70152d7640 1 -- start start 2024-03-29T11:09:14.422 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.443+0000 7f70152d7640 10 monclient: init 2024-03-29T11:09:14.423 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.445+0000 7f70152d7640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:14.423 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.445+0000 7f70152d7640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:09:14.423 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.445+0000 7f70152d7640 10 monclient(hunting): picked mon.noname-a con 0x7f70080389e0 addr v1:172.21.15.18:6789/0 2024-03-29T11:09:14.423 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.445+0000 7f70152d7640 10 monclient(hunting): picked mon.noname-b con 0x7f7008038dc0 addr v1:172.21.15.18:6790/0 2024-03-29T11:09:14.424 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.445+0000 7f70152d7640 10 monclient(hunting): picked mon.noname-c con 0x7f7008030660 addr v1:172.21.15.70:6789/0 2024-03-29T11:09:14.424 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.445+0000 7f70152d7640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70080c54c0 con 0x7f70080389e0 2024-03-29T11:09:14.424 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.445+0000 7f70152d7640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7008036430 con 0x7f7008030660 2024-03-29T11:09:14.424 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.445+0000 7f70152d7640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7008039840 con 0x7f7008038dc0 2024-03-29T11:09:14.424 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.445+0000 7f70152d7640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:14.424 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.445+0000 7f70152d7640 10 monclient(hunting): authenticate will time out at 780.717301s 2024-03-29T11:09:14.424 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.445+0000 7f7014ad6640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f7008030660 0x7f70080d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.18:44714/0 (socket says 172.21.15.18:44714) 2024-03-29T11:09:14.424 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.445+0000 7f7014ad6640 1 -- 172.21.15.18:0/1596054266 learned_addr learned my addr 172.21.15.18:0/1596054266 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:09:14.424 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.446+0000 7f700effd640 1 -- 172.21.15.18:0/1596054266 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 822016005 0 0) 0x7f7008036430 con 0x7f7008030660 2024-03-29T11:09:14.425 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.446+0000 7f700effd640 10 monclient(hunting): _negotiate 2024-03-29T11:09:14.425 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.446+0000 7f700effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:14.425 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.446+0000 7f700effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:14.425 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.446+0000 7f700effd640 10 monclient(hunting): my global_id is 4231 2024-03-29T11:09:14.425 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.446+0000 7f700effd640 1 -- 172.21.15.18:0/1596054266 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ff8003520 con 0x7f7008030660 2024-03-29T11:09:14.425 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.446+0000 7f700effd640 1 -- 172.21.15.18:0/1596054266 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1013360421 0 0) 0x7f70080c54c0 con 0x7f70080389e0 2024-03-29T11:09:14.426 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.447+0000 7f700effd640 10 monclient(hunting): _negotiate 2024-03-29T11:09:14.426 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.447+0000 7f700effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:14.426 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.447+0000 7f700effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:14.426 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.447+0000 7f700effd640 10 monclient(hunting): my global_id is 4236 2024-03-29T11:09:14.426 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.447+0000 7f700effd640 1 -- 172.21.15.18:0/1596054266 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7008036430 con 0x7f70080389e0 2024-03-29T11:09:14.426 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.447+0000 7f700effd640 1 -- 172.21.15.18:0/1596054266 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2747295519 0 0) 0x7f7008039840 con 0x7f7008038dc0 2024-03-29T11:09:14.426 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.447+0000 7f700effd640 10 monclient(hunting): _negotiate 2024-03-29T11:09:14.426 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.447+0000 7f700effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:14.426 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.447+0000 7f700effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:14.426 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.447+0000 7f700effd640 10 monclient(hunting): my global_id is 4238 2024-03-29T11:09:14.426 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.447+0000 7f700effd640 1 -- 172.21.15.18:0/1596054266 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ff80039d0 con 0x7f7008038dc0 2024-03-29T11:09:14.426 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.447+0000 7f700effd640 1 -- 172.21.15.18:0/1596054266 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1269825876 0 0) 0x7f6ff8003520 con 0x7f7008030660 2024-03-29T11:09:14.426 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.447+0000 7f700effd640 1 -- 172.21.15.18:0/1596054266 >> v1:172.21.15.18:6790/0 conn(0x7f7008038dc0 legacy=0x7f70080cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:14.426 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.447+0000 7f700effd640 1 -- 172.21.15.18:0/1596054266 >> v1:172.21.15.18:6789/0 conn(0x7f70080389e0 legacy=0x7f7008033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:14.427 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.447+0000 7f700effd640 10 monclient: _finish_hunting 0 2024-03-29T11:09:14.427 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.447+0000 7f700effd640 1 monclient: found mon.noname-c 2024-03-29T11:09:14.427 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.447+0000 7f700effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:14.427 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.447+0000 7f700effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.70:6789/0 2024-03-29T11:09:14.427 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.447+0000 7f700effd640 1 -- 172.21.15.18:0/1596054266 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7008035690 con 0x7f7008030660 2024-03-29T11:09:14.427 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.447+0000 7f700effd640 10 monclient: _finish_auth 0 2024-03-29T11:09:14.427 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.447+0000 7f700effd640 10 monclient: _check_auth_tickets 2024-03-29T11:09:14.427 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.447+0000 7f700effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:14.427 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.447+0000 7f70152d7640 5 monclient: authenticate success, global_id 4231 2024-03-29T11:09:14.427 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.447+0000 7f70152d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:14.427 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.447+0000 7f700effd640 1 -- 172.21.15.18:0/1596054266 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f701004f730 con 0x7f7008030660 2024-03-29T11:09:14.427 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.447+0000 7f700effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:14.427 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.447+0000 7f700effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:09:14.427 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.447+0000 7f700effd640 10 monclient: dump: 2024-03-29T11:09:14.428 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:09:14.428 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:14.428 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:14.428 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:14.428 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:14.428 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:14.428 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:14.428 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:14.428 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:14.428 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:14.428 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.447+0000 7f70152d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:14.428 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.447+0000 7f700effd640 1 -- 172.21.15.18:0/1596054266 <== mon.1 v1:172.21.15.70:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70100008d0 con 0x7f7008030660 2024-03-29T11:09:14.428 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.447+0000 7f700effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:14.428 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.447+0000 7f700effd640 1 -- 172.21.15.18:0/1596054266 <== mon.1 v1:172.21.15.70:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f70100758b0 con 0x7f7008030660 2024-03-29T11:09:14.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.447+0000 7f700effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:14.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.447+0000 7f700effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:09:14.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.447+0000 7f700effd640 10 monclient: dump: 2024-03-29T11:09:14.429 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:09:14.429 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:14.429 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:14.429 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:14.429 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:14.429 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:14.429 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:14.429 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:14.429 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:14.429 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:14.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.447+0000 7f70152d7640 10 monclient: get_monmap_and_config success 2024-03-29T11:09:14.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.447+0000 7f70152d7640 4 set_mon_vals no callback set 2024-03-29T11:09:14.431 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.448+0000 7f70152d7640 10 monclient: shutdown 2024-03-29T11:09:14.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.448+0000 7f70152d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:14.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.448+0000 7f70152d7640 1 -- 172.21.15.18:0/1596054266 >> v1:172.21.15.70:6789/0 conn(0x7f7008030660 legacy=0x7f70080d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:14.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.449+0000 7f70152d7640 1 -- 172.21.15.18:0/1596054266 shutdown_connections 2024-03-29T11:09:14.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.449+0000 7f70152d7640 1 -- 172.21.15.18:0/1596054266 >> 172.21.15.18:0/1596054266 conn(0x7f7008005c00 msgr2=0x7f7008006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:14.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.449+0000 7f70152d7640 1 -- 172.21.15.18:0/1596054266 shutdown_connections 2024-03-29T11:09:14.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.449+0000 7f70152d7640 1 -- 172.21.15.18:0/1596054266 wait complete. 2024-03-29T11:09:14.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.450+0000 7f70152d7640 10 monclient: build_initial_monmap 2024-03-29T11:09:14.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.450+0000 7f70152d7640 10 monclient: monmap: 2024-03-29T11:09:14.432 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:09:14.432 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:09:14.432 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:09:14.433 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:09:14.433 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:14.433 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:14.433 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:09:14.433 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:09:14.434 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:09:14.434 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:14.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.450+0000 7f70152d7640 1 librados: starting msgr at 2024-03-29T11:09:14.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.450+0000 7f70152d7640 1 librados: starting objecter 2024-03-29T11:09:14.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.450+0000 7f70152d7640 1 Processor -- start 2024-03-29T11:09:14.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.450+0000 7f70152d7640 1 -- start start 2024-03-29T11:09:14.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.450+0000 7f70152d7640 1 librados: setting wanted keys 2024-03-29T11:09:14.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.450+0000 7f70152d7640 1 librados: calling monclient init 2024-03-29T11:09:14.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.450+0000 7f70152d7640 10 monclient: init 2024-03-29T11:09:14.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.451+0000 7f70152d7640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:14.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.451+0000 7f70152d7640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:09:14.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.451+0000 7f70152d7640 10 monclient(hunting): picked mon.noname-b con 0x7f7008030660 addr v1:172.21.15.70:6789/0 2024-03-29T11:09:14.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.451+0000 7f70152d7640 10 monclient(hunting): picked mon.noname-c con 0x7f70080389e0 addr v1:172.21.15.18:6790/0 2024-03-29T11:09:14.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.451+0000 7f70152d7640 10 monclient(hunting): picked mon.noname-a con 0x7f70081618a0 addr v1:172.21.15.18:6789/0 2024-03-29T11:09:14.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.451+0000 7f70152d7640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7008161da0 con 0x7f70081618a0 2024-03-29T11:09:14.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.451+0000 7f70152d7640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f700800ce40 con 0x7f7008030660 2024-03-29T11:09:14.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.451+0000 7f70152d7640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f700800cfd0 con 0x7f70080389e0 2024-03-29T11:09:14.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.451+0000 7f70152d7640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:14.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.451+0000 7f70152d7640 10 monclient(hunting): authenticate will time out at 780.723873s 2024-03-29T11:09:14.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.452+0000 7f7014ad6640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f70081618a0 0x7f700802e190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:34738/0 (socket says 172.21.15.18:34738) 2024-03-29T11:09:14.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.452+0000 7f700ffff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f7008030660 0x7f7008161190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.18:44828/0 (socket says 172.21.15.18:44828) 2024-03-29T11:09:14.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.452+0000 7f700ffff640 1 -- 172.21.15.18:0/118969203 learned_addr learned my addr 172.21.15.18:0/118969203 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:09:14.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.452+0000 7f700cff9640 10 client.?.objecter ms_handle_connect 0x7f70081618a0 2024-03-29T11:09:14.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.452+0000 7f700cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:14.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.452+0000 7f700cff9640 10 client.?.objecter ms_handle_connect 0x7f7008030660 2024-03-29T11:09:14.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.452+0000 7f700cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:14.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.452+0000 7f700cff9640 10 client.?.objecter ms_handle_connect 0x7f70080389e0 2024-03-29T11:09:14.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.452+0000 7f700cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:14.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.453+0000 7f700cff9640 1 -- 172.21.15.18:0/118969203 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1529255139 0 0) 0x7f7008161da0 con 0x7f70081618a0 2024-03-29T11:09:14.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.453+0000 7f700cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:09:14.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.453+0000 7f700cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:14.437 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.453+0000 7f700cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:14.437 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.453+0000 7f700cff9640 10 monclient(hunting): my global_id is 4239 2024-03-29T11:09:14.437 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.453+0000 7f700cff9640 1 -- 172.21.15.18:0/118969203 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ffc004eb0 con 0x7f70081618a0 2024-03-29T11:09:14.437 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.453+0000 7f700cff9640 1 -- 172.21.15.18:0/118969203 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3137390976 0 0) 0x7f700800ce40 con 0x7f7008030660 2024-03-29T11:09:14.437 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.453+0000 7f700cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:09:14.437 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.453+0000 7f700cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:14.437 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.453+0000 7f700cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:14.437 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.453+0000 7f700cff9640 10 monclient(hunting): my global_id is 4234 2024-03-29T11:09:14.437 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.453+0000 7f700cff9640 1 -- 172.21.15.18:0/118969203 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7008161da0 con 0x7f7008030660 2024-03-29T11:09:14.437 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.453+0000 7f700cff9640 1 -- 172.21.15.18:0/118969203 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2406948385 0 0) 0x7f700800cfd0 con 0x7f70080389e0 2024-03-29T11:09:14.437 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.453+0000 7f700cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:09:14.437 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.453+0000 7f700cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:14.437 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.453+0000 7f700cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:14.437 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.453+0000 7f700cff9640 10 monclient(hunting): my global_id is 4241 2024-03-29T11:09:14.438 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.453+0000 7f700cff9640 1 -- 172.21.15.18:0/118969203 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f700800ce40 con 0x7f70080389e0 2024-03-29T11:09:14.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.454+0000 7f700cff9640 1 -- 172.21.15.18:0/118969203 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1226550820 0 0) 0x7f6ffc004eb0 con 0x7f70081618a0 2024-03-29T11:09:14.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.454+0000 7f700cff9640 1 -- 172.21.15.18:0/118969203 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f700800cfd0 con 0x7f70081618a0 2024-03-29T11:09:14.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.454+0000 7f700cff9640 1 -- 172.21.15.18:0/118969203 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f701004f400 con 0x7f70081618a0 2024-03-29T11:09:14.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.454+0000 7f700cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:14.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.454+0000 7f700cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:09:14.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.454+0000 7f700cff9640 10 monclient(hunting): dump: 2024-03-29T11:09:14.442 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:09:14.442 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:14.443 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:14.443 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:14.443 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:14.443 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:14.443 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:14.443 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:14.443 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:14.443 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:14.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.454+0000 7f700cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:09:14.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.454+0000 7f700cff9640 1 -- 172.21.15.18:0/118969203 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 334646138 0 0) 0x7f7008161da0 con 0x7f7008030660 2024-03-29T11:09:14.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.454+0000 7f700cff9640 1 -- 172.21.15.18:0/118969203 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ffc004eb0 con 0x7f7008030660 2024-03-29T11:09:14.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.454+0000 7f700cff9640 1 -- 172.21.15.18:0/118969203 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f7000001970 con 0x7f7008030660 2024-03-29T11:09:14.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.454+0000 7f700cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:14.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.454+0000 7f700cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:09:14.444 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.454+0000 7f700cff9640 10 monclient(hunting): dump: 2024-03-29T11:09:14.444 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:09:14.444 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:14.444 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:14.444 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:14.444 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:14.444 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:14.444 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:14.444 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:14.444 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:14.444 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:14.444 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.454+0000 7f700cff9640 1 -- 172.21.15.18:0/118969203 <== mon.1 v1:172.21.15.70:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2596785677 0 0) 0x7f6ffc004eb0 con 0x7f7008030660 2024-03-29T11:09:14.444 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.455+0000 7f700cff9640 1 -- 172.21.15.18:0/118969203 >> v1:172.21.15.18:6790/0 conn(0x7f70080389e0 legacy=0x7f700800c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:14.444 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.455+0000 7f700cff9640 1 -- 172.21.15.18:0/118969203 >> v1:172.21.15.18:6789/0 conn(0x7f70081618a0 legacy=0x7f700802e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:14.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.455+0000 7f700cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:09:14.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.455+0000 7f700cff9640 1 monclient: found mon.b 2024-03-29T11:09:14.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.455+0000 7f700cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:14.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.455+0000 7f700cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:14.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.455+0000 7f700cff9640 1 -- 172.21.15.18:0/118969203 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f700800d160 con 0x7f7008030660 2024-03-29T11:09:14.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.455+0000 7f700cff9640 10 monclient: _finish_auth 0 2024-03-29T11:09:14.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.455+0000 7f700cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:09:14.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.455+0000 7f700cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:14.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.455+0000 7f70152d7640 5 monclient: authenticate success, global_id 4234 2024-03-29T11:09:14.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.455+0000 7f70152d7640 10 monclient: _renew_subs 2024-03-29T11:09:14.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.455+0000 7f70152d7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:14.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.455+0000 7f70152d7640 1 -- 172.21.15.18:0/118969203 --> v1:172.21.15.70:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f700800c840 con 0x7f7008030660 2024-03-29T11:09:14.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.455+0000 7f70152d7640 10 client.4234.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:09:14.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.455+0000 7f70152d7640 10 monclient: _renew_subs 2024-03-29T11:09:14.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.455+0000 7f70152d7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:14.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.455+0000 7f70152d7640 1 -- 172.21.15.18:0/118969203 --> v1:172.21.15.70:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f700802e8a0 con 0x7f7008030660 2024-03-29T11:09:14.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.455+0000 7f70152d7640 1 librados: init done 2024-03-29T11:09:14.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.457+0000 7f700cff9640 1 -- 172.21.15.18:0/118969203 <== mon.1 v1:172.21.15.70:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7000001ad0 con 0x7f7008030660 2024-03-29T11:09:14.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.457+0000 7f700cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:14.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.457+0000 7f700cff9640 1 -- 172.21.15.18:0/118969203 <== mon.1 v1:172.21.15.70:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f7000003a40 con 0x7f7008030660 2024-03-29T11:09:14.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.457+0000 7f700cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:14.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.457+0000 7f700cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:09:14.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.457+0000 7f700cff9640 10 monclient: dump: 2024-03-29T11:09:14.446 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:09:14.446 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:14.446 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:14.446 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:14.446 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:14.446 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:14.447 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:14.447 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:14.447 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:14.447 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:14.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.457+0000 7f70152d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:14.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.457+0000 7f70152d7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:14.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.457+0000 7f70152d7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:14.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.457+0000 7f70152d7640 1 -- 172.21.15.18:0/118969203 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7008007b70 con 0x7f7008030660 2024-03-29T11:09:14.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.457+0000 7f700e7fc640 4 set_mon_vals no callback set 2024-03-29T11:09:14.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.458+0000 7f700cff9640 1 -- 172.21.15.18:0/118969203 <== mon.1 v1:172.21.15.70:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f7000003c60 con 0x7f7008030660 2024-03-29T11:09:14.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.459+0000 7f700cff9640 1 -- 172.21.15.18:0/118969203 <== mon.1 v1:172.21.15.70:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f7000084a60 con 0x7f7008030660 2024-03-29T11:09:14.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.459+0000 7f700cff9640 10 client.4234.objecter ms_dispatch 0x7f7010000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:09:14.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.459+0000 7f700cff9640 3 client.4234.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:09:14.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.459+0000 7f700cff9640 3 client.4234.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:09:14.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.460+0000 7f700cff9640 20 client.4234.objecter dump_active .. 0 homeless 2024-03-29T11:09:14.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.460+0000 7f700f7fe640 10 monclient: get_auth_request con 0x7f6ffc06de10 auth_method 0 2024-03-29T11:09:14.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.460+0000 7f700cff9640 10 client.4234.objecter ms_handle_connect 0x7f6ffc06de10 2024-03-29T11:09:14.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.464+0000 7f700cff9640 1 -- 172.21.15.18:0/118969203 <== mon.1 v1:172.21.15.70:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7000050340 con 0x7f7008030660 2024-03-29T11:09:14.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.464+0000 7f700cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:14.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.464+0000 7f700cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:09:14.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.892+0000 7f70152d7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:14.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.892+0000 7f70152d7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:14.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.892+0000 7f70152d7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:09:14.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.892+0000 7f70152d7640 1 -- 172.21.15.18:0/118969203 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7008005120 con 0x7f7008030660 2024-03-29T11:09:14.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.893+0000 7f700cff9640 1 -- 172.21.15.18:0/118969203 <== mon.1 v1:172.21.15.70:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f7000052270 con 0x7f7008030660 2024-03-29T11:09:14.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.893+0000 7f700cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:14.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.893+0000 7f700cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:09:14.872 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:09:14.872 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:09:14.873 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.895+0000 7f70152d7640 10 librados: watch_flush enter 2024-03-29T11:09:14.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.895+0000 7f70152d7640 10 librados: watch_flush exit 2024-03-29T11:09:14.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.895+0000 7f70152d7640 20 client.4234.objecter shutdown clearing up homeless session... 2024-03-29T11:09:14.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.895+0000 7f70152d7640 10 client.4234.objecter successfully canceled tick 2024-03-29T11:09:14.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.896+0000 7f70152d7640 1 -- 172.21.15.18:0/118969203 >> v1:172.21.15.70:6816/3101871562 conn(0x7f6ffc06de10 legacy=0x7f6ffc0702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:14.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.896+0000 7f70152d7640 10 monclient: shutdown 2024-03-29T11:09:14.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.896+0000 7f70152d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:14.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.896+0000 7f70152d7640 1 -- 172.21.15.18:0/118969203 >> v1:172.21.15.70:6789/0 conn(0x7f7008030660 legacy=0x7f7008161190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:14.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.896+0000 7f70152d7640 1 -- 172.21.15.18:0/118969203 shutdown_connections 2024-03-29T11:09:14.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.896+0000 7f70152d7640 1 -- 172.21.15.18:0/118969203 >> 172.21.15.18:0/118969203 conn(0x7f7008005c00 msgr2=0x7f7008005fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:14.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.896+0000 7f70152d7640 1 -- 172.21.15.18:0/118969203 shutdown_connections 2024-03-29T11:09:14.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.896+0000 7f70152d7640 1 -- 172.21.15.18:0/118969203 wait complete. 2024-03-29T11:09:14.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:14.897+0000 7f70152d7640 1 librados: shutdown 2024-03-29T11:09:14.889 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:09:21.696 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:09:21.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.874+0000 7f0ce8baf640 10 monclient: get_monmap_and_config 2024-03-29T11:09:21.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.874+0000 7f0ce8baf640 10 monclient: build_initial_monmap 2024-03-29T11:09:21.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.875+0000 7f0ce8baf640 10 monclient: monmap: 2024-03-29T11:09:21.853 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:09:21.854 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:21.854 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:09:21.875907+0000 2024-03-29T11:09:21.854 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:09:21.875907+0000 2024-03-29T11:09:21.854 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:21.854 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:21.854 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:09:21.854 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:09:21.854 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:09:21.854 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:21.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.875+0000 7f0ce8baf640 1 Processor -- start 2024-03-29T11:09:21.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.875+0000 7f0ce8baf640 1 -- start start 2024-03-29T11:09:21.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.875+0000 7f0ce8baf640 10 monclient: init 2024-03-29T11:09:21.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.877+0000 7f0ce8baf640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:21.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.877+0000 7f0ce8baf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:09:21.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.877+0000 7f0ce8baf640 10 monclient(hunting): picked mon.noname-a con 0x7f0ce40b0770 addr v1:172.21.15.18:6789/0 2024-03-29T11:09:21.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.877+0000 7f0ce8baf640 10 monclient(hunting): picked mon.noname-c con 0x7f0ce414a700 addr v1:172.21.15.70:6789/0 2024-03-29T11:09:21.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.877+0000 7f0ce8baf640 10 monclient(hunting): picked mon.noname-b con 0x7f0ce40ac5a0 addr v1:172.21.15.18:6790/0 2024-03-29T11:09:21.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.877+0000 7f0ce1d9b640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f0ce414a700 0x7f0ce40abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.18:41136/0 (socket says 172.21.15.18:41136) 2024-03-29T11:09:21.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.877+0000 7f0ce1d9b640 1 -- 172.21.15.18:0/664828447 learned_addr learned my addr 172.21.15.18:0/664828447 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:09:21.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.877+0000 7f0ce8baf640 1 -- 172.21.15.18:0/664828447 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce40aba70 con 0x7f0ce40b0770 2024-03-29T11:09:21.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.877+0000 7f0ce8baf640 1 -- 172.21.15.18:0/664828447 --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce40aecb0 con 0x7f0ce414a700 2024-03-29T11:09:21.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.877+0000 7f0ce8baf640 1 -- 172.21.15.18:0/664828447 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce40af150 con 0x7f0ce40ac5a0 2024-03-29T11:09:21.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.877+0000 7f0ce8baf640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:21.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.877+0000 7f0ce8baf640 10 monclient(hunting): authenticate will time out at 788.149867s 2024-03-29T11:09:21.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.878+0000 7f0ce159a640 1 -- 172.21.15.18:0/664828447 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 63460188 0 0) 0x7f0ce40af150 con 0x7f0ce40ac5a0 2024-03-29T11:09:21.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.878+0000 7f0ce159a640 10 monclient(hunting): _negotiate 2024-03-29T11:09:21.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.878+0000 7f0ce159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:21.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.878+0000 7f0ce159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:21.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.878+0000 7f0ce159a640 10 monclient(hunting): my global_id is 4244 2024-03-29T11:09:21.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.878+0000 7f0ce159a640 1 -- 172.21.15.18:0/664828447 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ccc003520 con 0x7f0ce40ac5a0 2024-03-29T11:09:21.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.878+0000 7f0ce159a640 1 -- 172.21.15.18:0/664828447 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1716112942 0 0) 0x7f0ce40aba70 con 0x7f0ce40b0770 2024-03-29T11:09:21.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.878+0000 7f0ce159a640 10 monclient(hunting): _negotiate 2024-03-29T11:09:21.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.878+0000 7f0ce159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:21.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.878+0000 7f0ce159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:21.858 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.878+0000 7f0ce159a640 10 monclient(hunting): my global_id is 4242 2024-03-29T11:09:21.858 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.878+0000 7f0ce159a640 1 -- 172.21.15.18:0/664828447 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ce40af150 con 0x7f0ce40b0770 2024-03-29T11:09:21.858 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.879+0000 7f0ce159a640 1 -- 172.21.15.18:0/664828447 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 91137395 0 0) 0x7f0ccc003520 con 0x7f0ce40ac5a0 2024-03-29T11:09:21.858 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.879+0000 7f0ce159a640 1 -- 172.21.15.18:0/664828447 >> v1:172.21.15.70:6789/0 conn(0x7f0ce414a700 legacy=0x7f0ce40abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:21.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.879+0000 7f0ce159a640 1 -- 172.21.15.18:0/664828447 >> v1:172.21.15.18:6789/0 conn(0x7f0ce40b0770 legacy=0x7f0ce4149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:21.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.879+0000 7f0ce159a640 10 monclient: _finish_hunting 0 2024-03-29T11:09:21.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.879+0000 7f0ce159a640 1 monclient: found mon.noname-b 2024-03-29T11:09:21.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.879+0000 7f0ce159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:21.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.879+0000 7f0ce159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:09:21.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.879+0000 7f0ce159a640 1 -- 172.21.15.18:0/664828447 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce40ac9d0 con 0x7f0ce40ac5a0 2024-03-29T11:09:21.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.879+0000 7f0ce159a640 10 monclient: _finish_auth 0 2024-03-29T11:09:21.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.879+0000 7f0ce159a640 10 monclient: _check_auth_tickets 2024-03-29T11:09:21.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.879+0000 7f0ce159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:21.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.879+0000 7f0ce8baf640 5 monclient: authenticate success, global_id 4244 2024-03-29T11:09:21.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.879+0000 7f0ce8baf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:21.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.879+0000 7f0ce159a640 1 -- 172.21.15.18:0/664828447 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f0cdc001a10 con 0x7f0ce40ac5a0 2024-03-29T11:09:21.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.879+0000 7f0ce159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:21.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.879+0000 7f0ce159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:09:21.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.879+0000 7f0ce159a640 10 monclient: dump: 2024-03-29T11:09:21.860 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:09:21.860 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:21.860 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:21.860 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:21.860 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:21.860 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:21.860 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:21.860 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:21.860 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:21.860 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:21.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.879+0000 7f0ce8baf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:21.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.880+0000 7f0ce159a640 1 -- 172.21.15.18:0/664828447 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0cdc001c30 con 0x7f0ce40ac5a0 2024-03-29T11:09:21.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.880+0000 7f0ce159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:21.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.880+0000 7f0ce159a640 1 -- 172.21.15.18:0/664828447 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f0cdc001fc0 con 0x7f0ce40ac5a0 2024-03-29T11:09:21.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.880+0000 7f0ce8baf640 10 monclient: get_monmap_and_config success 2024-03-29T11:09:21.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.880+0000 7f0ce8baf640 4 set_mon_vals no callback set 2024-03-29T11:09:21.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.880+0000 7f0ce159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:21.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.881+0000 7f0ce159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:09:21.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.881+0000 7f0ce159a640 10 monclient: dump: 2024-03-29T11:09:21.863 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:09:21.863 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:21.863 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:21.863 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:21.863 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:21.863 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:21.863 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:21.863 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:21.863 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:21.863 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:21.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.881+0000 7f0ce8baf640 10 monclient: shutdown 2024-03-29T11:09:21.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.881+0000 7f0ce8baf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:21.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.881+0000 7f0ce8baf640 1 -- 172.21.15.18:0/664828447 >> v1:172.21.15.18:6790/0 conn(0x7f0ce40ac5a0 legacy=0x7f0ce4151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:21.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.881+0000 7f0ce8baf640 1 -- 172.21.15.18:0/664828447 shutdown_connections 2024-03-29T11:09:21.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.881+0000 7f0ce8baf640 1 -- 172.21.15.18:0/664828447 >> 172.21.15.18:0/664828447 conn(0x7f0ce406b660 msgr2=0x7f0ce406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:21.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.881+0000 7f0ce8baf640 1 -- 172.21.15.18:0/664828447 shutdown_connections 2024-03-29T11:09:21.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.881+0000 7f0ce8baf640 1 -- 172.21.15.18:0/664828447 wait complete. 2024-03-29T11:09:21.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.882+0000 7f0ce8baf640 10 monclient: build_initial_monmap 2024-03-29T11:09:21.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.883+0000 7f0ce8baf640 10 monclient: monmap: 2024-03-29T11:09:21.864 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:09:21.864 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:09:21.864 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:09:21.864 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:09:21.864 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:21.864 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:21.865 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:09:21.865 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:09:21.865 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:09:21.865 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:21.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.883+0000 7f0ce8baf640 1 librados: starting msgr at 2024-03-29T11:09:21.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.883+0000 7f0ce8baf640 1 librados: starting objecter 2024-03-29T11:09:21.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.884+0000 7f0ce8baf640 1 Processor -- start 2024-03-29T11:09:21.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.884+0000 7f0ce8baf640 1 -- start start 2024-03-29T11:09:21.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.884+0000 7f0ce8baf640 1 librados: setting wanted keys 2024-03-29T11:09:21.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.884+0000 7f0ce8baf640 1 librados: calling monclient init 2024-03-29T11:09:21.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.884+0000 7f0ce8baf640 10 monclient: init 2024-03-29T11:09:21.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.885+0000 7f0ce8baf640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:21.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.885+0000 7f0ce8baf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:09:21.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.885+0000 7f0ce8baf640 10 monclient(hunting): picked mon.noname-a con 0x7f0ce40ac5a0 addr v1:172.21.15.18:6789/0 2024-03-29T11:09:21.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.886+0000 7f0ce259c640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f0ce40ac5a0 0x7f0ce41dbb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:42998/0 (socket says 172.21.15.18:42998) 2024-03-29T11:09:21.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.886+0000 7f0ce8baf640 10 monclient(hunting): picked mon.noname-c con 0x7f0ce40b0770 addr v1:172.21.15.18:6790/0 2024-03-29T11:09:21.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.886+0000 7f0ce8baf640 10 monclient(hunting): picked mon.noname-b con 0x7f0ce414a700 addr v1:172.21.15.70:6789/0 2024-03-29T11:09:21.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.886+0000 7f0ce8baf640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce41dc310 con 0x7f0ce40ac5a0 2024-03-29T11:09:21.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.886+0000 7f0ce8baf640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce4071c10 con 0x7f0ce414a700 2024-03-29T11:09:21.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.886+0000 7f0ce1d9b640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f0ce40b0770 0x7f0ce4144150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:43664/0 (socket says 172.21.15.18:43664) 2024-03-29T11:09:21.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.886+0000 7f0ce1d9b640 1 -- 172.21.15.18:0/3722749645 learned_addr learned my addr 172.21.15.18:0/3722749645 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:09:21.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.886+0000 7f0ce8baf640 1 -- 172.21.15.18:0/3722749645 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ce41d91e0 con 0x7f0ce40b0770 2024-03-29T11:09:21.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.886+0000 7f0cd2ffd640 10 client.?.objecter ms_handle_connect 0x7f0ce40ac5a0 2024-03-29T11:09:21.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.886+0000 7f0cd2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:21.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.887+0000 7f0cd2ffd640 10 client.?.objecter ms_handle_connect 0x7f0ce40b0770 2024-03-29T11:09:21.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.887+0000 7f0cd2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:21.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.887+0000 7f0ce8baf640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:21.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.887+0000 7f0ce8baf640 10 monclient(hunting): authenticate will time out at 788.159002s 2024-03-29T11:09:21.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.887+0000 7f0cd2ffd640 10 client.?.objecter ms_handle_connect 0x7f0ce414a700 2024-03-29T11:09:21.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.887+0000 7f0cd2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:21.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.887+0000 7f0cd2ffd640 1 -- 172.21.15.18:0/3722749645 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1334858851 0 0) 0x7f0ce41dc310 con 0x7f0ce40ac5a0 2024-03-29T11:09:21.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.887+0000 7f0cd2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:09:21.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.887+0000 7f0cd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:21.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.887+0000 7f0cd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:21.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.887+0000 7f0cd2ffd640 10 monclient(hunting): my global_id is 4245 2024-03-29T11:09:21.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.887+0000 7f0cd2ffd640 1 -- 172.21.15.18:0/3722749645 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cc4004eb0 con 0x7f0ce40ac5a0 2024-03-29T11:09:21.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.887+0000 7f0cd2ffd640 1 -- 172.21.15.18:0/3722749645 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3863159066 0 0) 0x7f0ce41d91e0 con 0x7f0ce40b0770 2024-03-29T11:09:21.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.887+0000 7f0cd2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:09:21.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.887+0000 7f0cd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:21.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.887+0000 7f0cd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:21.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.887+0000 7f0cd2ffd640 10 monclient(hunting): my global_id is 4247 2024-03-29T11:09:21.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.887+0000 7f0cd2ffd640 1 -- 172.21.15.18:0/3722749645 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ce41dc310 con 0x7f0ce40b0770 2024-03-29T11:09:21.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.887+0000 7f0cd2ffd640 1 -- 172.21.15.18:0/3722749645 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2453071809 0 0) 0x7f0ce4071c10 con 0x7f0ce414a700 2024-03-29T11:09:21.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.887+0000 7f0cd2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:09:21.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.887+0000 7f0cd2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:21.870 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.887+0000 7f0cd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:21.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.887+0000 7f0cd2ffd640 10 monclient(hunting): my global_id is 4237 2024-03-29T11:09:21.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.887+0000 7f0cd2ffd640 1 -- 172.21.15.18:0/3722749645 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ce41d91e0 con 0x7f0ce414a700 2024-03-29T11:09:21.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.888+0000 7f0cd2ffd640 1 -- 172.21.15.18:0/3722749645 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 976987888 0 0) 0x7f0cc4004eb0 con 0x7f0ce40ac5a0 2024-03-29T11:09:21.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.888+0000 7f0cd2ffd640 1 -- 172.21.15.18:0/3722749645 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ce4071c10 con 0x7f0ce40ac5a0 2024-03-29T11:09:21.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.888+0000 7f0cd2ffd640 1 -- 172.21.15.18:0/3722749645 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f0cd40019b0 con 0x7f0ce40ac5a0 2024-03-29T11:09:21.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.888+0000 7f0cd2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:21.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.888+0000 7f0cd2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:09:21.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.888+0000 7f0cd2ffd640 10 monclient(hunting): dump: 2024-03-29T11:09:21.871 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:09:21.871 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:21.871 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:21.871 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:21.871 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:21.871 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:21.872 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:21.872 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:21.872 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:21.872 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:21.872 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.888+0000 7f0cd2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:09:21.872 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.888+0000 7f0cd2ffd640 1 -- 172.21.15.18:0/3722749645 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4192848467 0 0) 0x7f0ce4071c10 con 0x7f0ce40ac5a0 2024-03-29T11:09:21.872 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.888+0000 7f0cd2ffd640 1 -- 172.21.15.18:0/3722749645 >> v1:172.21.15.18:6790/0 conn(0x7f0ce40b0770 legacy=0x7f0ce4144150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:21.872 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.888+0000 7f0cd2ffd640 1 -- 172.21.15.18:0/3722749645 >> v1:172.21.15.70:6789/0 conn(0x7f0ce414a700 legacy=0x7f0ce4123ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:21.872 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.888+0000 7f0cd2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:09:21.872 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.888+0000 7f0cd2ffd640 1 monclient: found mon.a 2024-03-29T11:09:21.872 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.888+0000 7f0cd2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:21.873 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.888+0000 7f0cd2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:09:21.873 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.888+0000 7f0cd2ffd640 1 -- 172.21.15.18:0/3722749645 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce4070e40 con 0x7f0ce40ac5a0 2024-03-29T11:09:21.873 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.888+0000 7f0cd2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:09:21.873 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.888+0000 7f0cd2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:09:21.873 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.888+0000 7f0cd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:21.873 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.888+0000 7f0ce8baf640 5 monclient: authenticate success, global_id 4245 2024-03-29T11:09:21.873 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.888+0000 7f0ce8baf640 10 monclient: _renew_subs 2024-03-29T11:09:21.873 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.888+0000 7f0ce8baf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:09:21.873 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.888+0000 7f0ce8baf640 1 -- 172.21.15.18:0/3722749645 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ce40710f0 con 0x7f0ce40ac5a0 2024-03-29T11:09:21.873 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.888+0000 7f0ce8baf640 10 client.4245.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:09:21.873 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.888+0000 7f0ce8baf640 10 monclient: _renew_subs 2024-03-29T11:09:21.873 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.888+0000 7f0ce8baf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:09:21.873 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.888+0000 7f0ce8baf640 1 -- 172.21.15.18:0/3722749645 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ce4144a40 con 0x7f0ce40ac5a0 2024-03-29T11:09:21.873 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.888+0000 7f0ce8baf640 1 librados: init done 2024-03-29T11:09:21.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.889+0000 7f0cd2ffd640 1 -- 172.21.15.18:0/3722749645 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0cd40035f0 con 0x7f0ce40ac5a0 2024-03-29T11:09:21.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.889+0000 7f0cd2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:21.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.889+0000 7f0cd2ffd640 1 -- 172.21.15.18:0/3722749645 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f0cd4003980 con 0x7f0ce40ac5a0 2024-03-29T11:09:21.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.889+0000 7f0cd2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:21.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.889+0000 7f0cd2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:09:21.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.889+0000 7f0cd2ffd640 10 monclient: dump: 2024-03-29T11:09:21.874 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:09:21.874 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:21.874 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:21.874 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:21.874 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:21.874 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:21.874 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:21.874 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:21.874 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:21.875 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:21.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.889+0000 7f0cd3fff640 4 set_mon_vals no callback set 2024-03-29T11:09:21.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.890+0000 7f0ce8baf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:21.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.890+0000 7f0ce8baf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:21.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.890+0000 7f0ce8baf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:09:21.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.890+0000 7f0ce8baf640 1 -- 172.21.15.18:0/3722749645 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cc8005080 con 0x7f0ce40ac5a0 2024-03-29T11:09:21.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.891+0000 7f0cd2ffd640 1 -- 172.21.15.18:0/3722749645 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f0cd4003ae0 con 0x7f0ce40ac5a0 2024-03-29T11:09:21.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.892+0000 7f0ce1d9b640 10 monclient: get_auth_request con 0x7f0cc406dc30 auth_method 0 2024-03-29T11:09:21.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.892+0000 7f0cd2ffd640 1 -- 172.21.15.18:0/3722749645 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f0cd4084a70 con 0x7f0ce40ac5a0 2024-03-29T11:09:21.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.892+0000 7f0cd2ffd640 10 client.4245.objecter ms_dispatch 0x7f0ce4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:09:21.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.892+0000 7f0cd2ffd640 3 client.4245.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:09:21.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.892+0000 7f0cd2ffd640 3 client.4245.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:09:21.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.892+0000 7f0cd2ffd640 20 client.4245.objecter dump_active .. 0 homeless 2024-03-29T11:09:21.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.892+0000 7f0cd2ffd640 10 client.4245.objecter ms_handle_connect 0x7f0cc406dc30 2024-03-29T11:09:21.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.900+0000 7f0cd2ffd640 1 -- 172.21.15.18:0/3722749645 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0cd4050350 con 0x7f0ce40ac5a0 2024-03-29T11:09:21.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.900+0000 7f0cd2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:21.879 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:21.900+0000 7f0cd2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:09:22.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:22.221+0000 7f0ce8baf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:22.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:22.221+0000 7f0ce8baf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:22.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:22.221+0000 7f0ce8baf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:09:22.200 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:22.221+0000 7f0ce8baf640 1 -- 172.21.15.18:0/3722749645 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0cc8005330 con 0x7f0ce40ac5a0 2024-03-29T11:09:22.200 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:22.221+0000 7f0cd2ffd640 1 -- 172.21.15.18:0/3722749645 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f0cd4052280 con 0x7f0ce40ac5a0 2024-03-29T11:09:22.200 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:22.221+0000 7f0cd2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:22.200 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:22.221+0000 7f0cd2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:09:22.200 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:09:22.200 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:09:22.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:22.225+0000 7f0ce8baf640 10 librados: watch_flush enter 2024-03-29T11:09:22.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:22.225+0000 7f0ce8baf640 10 librados: watch_flush exit 2024-03-29T11:09:22.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:22.225+0000 7f0ce8baf640 20 client.4245.objecter shutdown clearing up homeless session... 2024-03-29T11:09:22.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:22.225+0000 7f0ce8baf640 10 client.4245.objecter successfully canceled tick 2024-03-29T11:09:22.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:22.225+0000 7f0ce8baf640 1 -- 172.21.15.18:0/3722749645 >> v1:172.21.15.70:6816/3101871562 conn(0x7f0cc406dc30 legacy=0x7f0cc40700f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:22.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:22.225+0000 7f0ce8baf640 10 monclient: shutdown 2024-03-29T11:09:22.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:22.225+0000 7f0ce8baf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:22.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:22.225+0000 7f0ce8baf640 1 -- 172.21.15.18:0/3722749645 >> v1:172.21.15.18:6789/0 conn(0x7f0ce40ac5a0 legacy=0x7f0ce41dbb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:22.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:22.225+0000 7f0ce8baf640 1 -- 172.21.15.18:0/3722749645 shutdown_connections 2024-03-29T11:09:22.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:22.225+0000 7f0ce8baf640 1 -- 172.21.15.18:0/3722749645 >> 172.21.15.18:0/3722749645 conn(0x7f0ce406b660 msgr2=0x7f0ce4149580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:22.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:22.225+0000 7f0ce8baf640 1 -- 172.21.15.18:0/3722749645 shutdown_connections 2024-03-29T11:09:22.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:22.225+0000 7f0ce8baf640 1 -- 172.21.15.18:0/3722749645 wait complete. 2024-03-29T11:09:22.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:22.225+0000 7f0ce8baf640 1 librados: shutdown 2024-03-29T11:09:22.217 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:09:29.025 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:09:29.166 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.188+0000 7f0fbecd6640 10 monclient: get_monmap_and_config 2024-03-29T11:09:29.167 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.188+0000 7f0fbecd6640 10 monclient: build_initial_monmap 2024-03-29T11:09:29.167 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.188+0000 7f0fbecd6640 10 monclient: monmap: 2024-03-29T11:09:29.167 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:09:29.167 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:29.167 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:09:29.189481+0000 2024-03-29T11:09:29.167 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:09:29.189481+0000 2024-03-29T11:09:29.167 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:29.167 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:29.167 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:09:29.168 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:09:29.168 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:09:29.168 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:29.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.189+0000 7f0fbecd6640 1 Processor -- start 2024-03-29T11:09:29.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.190+0000 7f0fbecd6640 1 -- start start 2024-03-29T11:09:29.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.190+0000 7f0fbecd6640 10 monclient: init 2024-03-29T11:09:29.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.190+0000 7f0fbecd6640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:29.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.190+0000 7f0fbecd6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:09:29.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.190+0000 7f0fbecd6640 10 monclient(hunting): picked mon.noname-b con 0x7f0fb80b1f60 addr v1:172.21.15.18:6790/0 2024-03-29T11:09:29.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.190+0000 7f0fbecd6640 10 monclient(hunting): picked mon.noname-c con 0x7f0fb8071eb0 addr v1:172.21.15.70:6789/0 2024-03-29T11:09:29.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.193+0000 7f0fbca72640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f0fb80b1f60 0x7f0fb8143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:38311/0 (socket says 172.21.15.18:38311) 2024-03-29T11:09:29.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.193+0000 7f0fb7fff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f0fb8071eb0 0x7f0fb8146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.18:60240/0 (socket says 172.21.15.18:60240) 2024-03-29T11:09:29.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.193+0000 7f0fbd273640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f0fb8143d10 0x7f0fb8146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:57373/0 (socket says 172.21.15.18:57373) 2024-03-29T11:09:29.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.193+0000 7f0fbca72640 1 -- 172.21.15.18:0/3858595764 learned_addr learned my addr 172.21.15.18:0/3858595764 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:09:29.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.193+0000 7f0fbecd6640 10 monclient(hunting): picked mon.noname-a con 0x7f0fb8143d10 addr v1:172.21.15.18:6789/0 2024-03-29T11:09:29.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.193+0000 7f0fbecd6640 1 -- 172.21.15.18:0/3858595764 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fb806ef80 con 0x7f0fb8143d10 2024-03-29T11:09:29.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.193+0000 7f0fbecd6640 1 -- 172.21.15.18:0/3858595764 --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fb806f260 con 0x7f0fb8071eb0 2024-03-29T11:09:29.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.193+0000 7f0fbecd6640 1 -- 172.21.15.18:0/3858595764 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fb80b40c0 con 0x7f0fb80b1f60 2024-03-29T11:09:29.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.193+0000 7f0fbecd6640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:29.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.193+0000 7f0fbecd6640 10 monclient(hunting): authenticate will time out at 795.465700s 2024-03-29T11:09:29.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.194+0000 7f0fb77fe640 1 -- 172.21.15.18:0/3858595764 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4219403132 0 0) 0x7f0fb806ef80 con 0x7f0fb8143d10 2024-03-29T11:09:29.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.194+0000 7f0fb77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:09:29.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.194+0000 7f0fb77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:29.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.194+0000 7f0fb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:29.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.194+0000 7f0fb77fe640 10 monclient(hunting): my global_id is 4248 2024-03-29T11:09:29.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.194+0000 7f0fb77fe640 1 -- 172.21.15.18:0/3858595764 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fa0003520 con 0x7f0fb8143d10 2024-03-29T11:09:29.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.194+0000 7f0fb77fe640 1 -- 172.21.15.18:0/3858595764 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 651001504 0 0) 0x7f0fb806f260 con 0x7f0fb8071eb0 2024-03-29T11:09:29.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.194+0000 7f0fb77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:09:29.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.194+0000 7f0fb77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:29.175 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.194+0000 7f0fb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:29.175 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.194+0000 7f0fb77fe640 10 monclient(hunting): my global_id is 4240 2024-03-29T11:09:29.175 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.194+0000 7f0fb77fe640 1 -- 172.21.15.18:0/3858595764 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fb806ef80 con 0x7f0fb8071eb0 2024-03-29T11:09:29.175 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.194+0000 7f0fb77fe640 1 -- 172.21.15.18:0/3858595764 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 689055671 0 0) 0x7f0fa0003520 con 0x7f0fb8143d10 2024-03-29T11:09:29.175 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.194+0000 7f0fb77fe640 1 -- 172.21.15.18:0/3858595764 >> v1:172.21.15.18:6790/0 conn(0x7f0fb80b1f60 legacy=0x7f0fb8143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:29.175 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.195+0000 7f0fb77fe640 1 -- 172.21.15.18:0/3858595764 >> v1:172.21.15.70:6789/0 conn(0x7f0fb8071eb0 legacy=0x7f0fb8146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:29.175 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.195+0000 7f0fb77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:09:29.175 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.195+0000 7f0fb77fe640 1 monclient: found mon.noname-a 2024-03-29T11:09:29.175 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.195+0000 7f0fb77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:29.175 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.195+0000 7f0fb77fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-29T11:09:29.175 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.195+0000 7f0fb77fe640 1 -- 172.21.15.18:0/3858595764 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb8155d70 con 0x7f0fb8143d10 2024-03-29T11:09:29.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.195+0000 7f0fb77fe640 10 monclient: _finish_auth 0 2024-03-29T11:09:29.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.195+0000 7f0fb77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:09:29.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.195+0000 7f0fb77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:29.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.195+0000 7f0fb77fe640 1 -- 172.21.15.18:0/3858595764 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f0fb0001990 con 0x7f0fb8143d10 2024-03-29T11:09:29.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.195+0000 7f0fb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:29.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.195+0000 7f0fb77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:09:29.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.195+0000 7f0fb77fe640 10 monclient: dump: 2024-03-29T11:09:29.176 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:09:29.176 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:29.176 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:29.176 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:29.176 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:29.176 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:29.177 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:29.177 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:29.177 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:29.177 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:29.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.195+0000 7f0fbecd6640 5 monclient: authenticate success, global_id 4248 2024-03-29T11:09:29.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.195+0000 7f0fbecd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:29.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.198+0000 7f0fb77fe640 1 -- 172.21.15.18:0/3858595764 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0fb0001af0 con 0x7f0fb8143d10 2024-03-29T11:09:29.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.198+0000 7f0fb77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:29.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.198+0000 7f0fb77fe640 1 -- 172.21.15.18:0/3858595764 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f0fb0001f00 con 0x7f0fb8143d10 2024-03-29T11:09:29.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.198+0000 7f0fb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:29.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.198+0000 7f0fb77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:09:29.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.198+0000 7f0fb77fe640 10 monclient: dump: 2024-03-29T11:09:29.178 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:09:29.178 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:29.178 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:29.179 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:29.179 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:29.179 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:29.179 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:29.179 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:29.179 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:29.179 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:29.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.198+0000 7f0fbecd6640 10 monclient: get_monmap_and_config success 2024-03-29T11:09:29.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.198+0000 7f0fbecd6640 4 set_mon_vals no callback set 2024-03-29T11:09:29.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.199+0000 7f0fbecd6640 10 monclient: shutdown 2024-03-29T11:09:29.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.199+0000 7f0fbecd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:29.180 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.199+0000 7f0fbecd6640 1 -- 172.21.15.18:0/3858595764 >> v1:172.21.15.18:6789/0 conn(0x7f0fb8143d10 legacy=0x7f0fb8146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:29.180 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.199+0000 7f0fbecd6640 1 -- 172.21.15.18:0/3858595764 shutdown_connections 2024-03-29T11:09:29.180 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.199+0000 7f0fbecd6640 1 -- 172.21.15.18:0/3858595764 >> 172.21.15.18:0/3858595764 conn(0x7f0fb8066a00 msgr2=0x7f0fb8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:29.180 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.199+0000 7f0fbecd6640 1 -- 172.21.15.18:0/3858595764 shutdown_connections 2024-03-29T11:09:29.180 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.199+0000 7f0fbecd6640 1 -- 172.21.15.18:0/3858595764 wait complete. 2024-03-29T11:09:29.180 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.200+0000 7f0fbecd6640 10 monclient: build_initial_monmap 2024-03-29T11:09:29.180 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.200+0000 7f0fbecd6640 10 monclient: monmap: 2024-03-29T11:09:29.181 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:09:29.181 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:09:29.181 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:09:29.181 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:09:29.181 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:29.181 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:29.181 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:09:29.181 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:09:29.181 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:09:29.181 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:29.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.200+0000 7f0fbecd6640 1 librados: starting msgr at 2024-03-29T11:09:29.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.200+0000 7f0fbecd6640 1 librados: starting objecter 2024-03-29T11:09:29.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.201+0000 7f0fbecd6640 1 Processor -- start 2024-03-29T11:09:29.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.203+0000 7f0fbecd6640 1 -- start start 2024-03-29T11:09:29.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.203+0000 7f0fbecd6640 1 librados: setting wanted keys 2024-03-29T11:09:29.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.203+0000 7f0fbecd6640 1 librados: calling monclient init 2024-03-29T11:09:29.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.203+0000 7f0fbecd6640 10 monclient: init 2024-03-29T11:09:29.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.204+0000 7f0fbecd6640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:29.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.204+0000 7f0fbecd6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:09:29.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.204+0000 7f0fbecd6640 10 monclient(hunting): picked mon.noname-b con 0x7f0fb80b1f60 addr v1:172.21.15.70:6789/0 2024-03-29T11:09:29.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.204+0000 7f0fbecd6640 10 monclient(hunting): picked mon.noname-a con 0x7f0fb8143d10 addr v1:172.21.15.18:6789/0 2024-03-29T11:09:29.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.204+0000 7f0fbecd6640 10 monclient(hunting): picked mon.noname-c con 0x7f0fb80a94e0 addr v1:172.21.15.18:6790/0 2024-03-29T11:09:29.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.204+0000 7f0fbecd6640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fb813cae0 con 0x7f0fb8143d10 2024-03-29T11:09:29.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.204+0000 7f0fbecd6640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fb813d270 con 0x7f0fb80b1f60 2024-03-29T11:09:29.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.204+0000 7f0fbecd6640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fb80acef0 con 0x7f0fb80a94e0 2024-03-29T11:09:29.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.204+0000 7f0fbecd6640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:29.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.204+0000 7f0fbecd6640 10 monclient(hunting): authenticate will time out at 795.476391s 2024-03-29T11:09:29.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.208+0000 7f0fbca72640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f0fb80b1f60 0x7f0fb80708d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.18:60440/0 (socket says 172.21.15.18:60440) 2024-03-29T11:09:29.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.208+0000 7f0fbca72640 1 -- 172.21.15.18:0/881212469 learned_addr learned my addr 172.21.15.18:0/881212469 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:09:29.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.208+0000 7f0fb57fa640 10 client.?.objecter ms_handle_connect 0x7f0fb8143d10 2024-03-29T11:09:29.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.208+0000 7f0fb57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:29.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.208+0000 7f0fb57fa640 1 -- 172.21.15.18:0/881212469 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1737864127 0 0) 0x7f0fb813cae0 con 0x7f0fb8143d10 2024-03-29T11:09:29.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.208+0000 7f0fb57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:09:29.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.209+0000 7f0fb57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:29.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.209+0000 7f0fb57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:29.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.209+0000 7f0fb57fa640 10 monclient(hunting): my global_id is 4251 2024-03-29T11:09:29.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.209+0000 7f0fb57fa640 1 -- 172.21.15.18:0/881212469 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fa4004eb0 con 0x7f0fb8143d10 2024-03-29T11:09:29.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.209+0000 7f0fb57fa640 10 client.?.objecter ms_handle_connect 0x7f0fb80b1f60 2024-03-29T11:09:29.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.209+0000 7f0fb57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:29.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.209+0000 7f0fb57fa640 1 -- 172.21.15.18:0/881212469 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3614621119 0 0) 0x7f0fb813d270 con 0x7f0fb80b1f60 2024-03-29T11:09:29.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.210+0000 7f0fb57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:09:29.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.210+0000 7f0fb57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:29.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.210+0000 7f0fb57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:29.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.210+0000 7f0fb57fa640 10 monclient(hunting): my global_id is 4243 2024-03-29T11:09:29.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.210+0000 7f0fb57fa640 1 -- 172.21.15.18:0/881212469 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fb813cae0 con 0x7f0fb80b1f60 2024-03-29T11:09:29.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.210+0000 7f0fb57fa640 1 -- 172.21.15.18:0/881212469 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 371231621 0 0) 0x7f0fa4004eb0 con 0x7f0fb8143d10 2024-03-29T11:09:29.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.210+0000 7f0fb57fa640 1 -- 172.21.15.18:0/881212469 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0fb813d270 con 0x7f0fb8143d10 2024-03-29T11:09:29.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.210+0000 7f0fb57fa640 1 -- 172.21.15.18:0/881212469 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f0fac001860 con 0x7f0fb8143d10 2024-03-29T11:09:29.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.210+0000 7f0fb57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:29.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.210+0000 7f0fb57fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:09:29.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.210+0000 7f0fb57fa640 10 monclient(hunting): dump: 2024-03-29T11:09:29.195 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:09:29.196 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:29.196 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:29.196 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:29.196 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:29.196 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:29.196 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:29.196 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:29.196 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:29.196 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:29.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.210+0000 7f0fb57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:09:29.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.211+0000 7f0fb57fa640 1 -- 172.21.15.18:0/881212469 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 865750231 0 0) 0x7f0fb813cae0 con 0x7f0fb80b1f60 2024-03-29T11:09:29.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.211+0000 7f0fb57fa640 1 -- 172.21.15.18:0/881212469 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0fa4004eb0 con 0x7f0fb80b1f60 2024-03-29T11:09:29.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.211+0000 7f0fb57fa640 1 -- 172.21.15.18:0/881212469 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1227758485 0 0) 0x7f0fb813d270 con 0x7f0fb8143d10 2024-03-29T11:09:29.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.211+0000 7f0fb57fa640 1 -- 172.21.15.18:0/881212469 >> v1:172.21.15.18:6790/0 conn(0x7f0fb80a94e0 legacy=0x7f0fb80a9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:29.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.211+0000 7f0fb57fa640 1 -- 172.21.15.18:0/881212469 >> v1:172.21.15.70:6789/0 conn(0x7f0fb80b1f60 legacy=0x7f0fb80708d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:29.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.211+0000 7f0fb57fa640 10 monclient: _finish_hunting 0 2024-03-29T11:09:29.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.211+0000 7f0fb57fa640 1 monclient: found mon.a 2024-03-29T11:09:29.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.211+0000 7f0fb57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:29.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.211+0000 7f0fb57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:09:29.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.211+0000 7f0fb57fa640 1 -- 172.21.15.18:0/881212469 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb813cd70 con 0x7f0fb8143d10 2024-03-29T11:09:29.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.211+0000 7f0fb57fa640 10 monclient: _finish_auth 0 2024-03-29T11:09:29.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.211+0000 7f0fb57fa640 10 monclient: _check_auth_tickets 2024-03-29T11:09:29.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.211+0000 7f0fb57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:29.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.211+0000 7f0fb57fa640 10 client.?.objecter ms_handle_connect 0x7f0fb80a94e0 2024-03-29T11:09:29.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.211+0000 7f0fb57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:29.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.211+0000 7f0fbecd6640 5 monclient: authenticate success, global_id 4251 2024-03-29T11:09:29.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.211+0000 7f0fbecd6640 10 monclient: _renew_subs 2024-03-29T11:09:29.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.211+0000 7f0fbecd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:09:29.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.211+0000 7f0fbecd6640 1 -- 172.21.15.18:0/881212469 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fb80aa670 con 0x7f0fb8143d10 2024-03-29T11:09:29.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.211+0000 7f0fbecd6640 10 client.4251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:09:29.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.211+0000 7f0fbecd6640 10 monclient: _renew_subs 2024-03-29T11:09:29.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.211+0000 7f0fbecd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:09:29.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.211+0000 7f0fbecd6640 1 -- 172.21.15.18:0/881212469 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fb81dee20 con 0x7f0fb8143d10 2024-03-29T11:09:29.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.211+0000 7f0fbecd6640 1 librados: init done 2024-03-29T11:09:29.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.211+0000 7f0fb57fa640 1 -- 172.21.15.18:0/881212469 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0fac0019c0 con 0x7f0fb8143d10 2024-03-29T11:09:29.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.211+0000 7f0fb57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:29.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.211+0000 7f0fb67fc640 4 set_mon_vals no callback set 2024-03-29T11:09:29.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.212+0000 7f0fb57fa640 1 -- 172.21.15.18:0/881212469 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f0fac004980 con 0x7f0fb8143d10 2024-03-29T11:09:29.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.212+0000 7f0fb57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:29.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.212+0000 7f0fb57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:09:29.198 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.212+0000 7f0fb57fa640 10 monclient: dump: 2024-03-29T11:09:29.198 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:09:29.199 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:29.199 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:29.199 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:29.199 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:29.199 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:29.199 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:29.199 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:29.199 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:29.199 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:29.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.213+0000 7f0fbecd6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:29.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.213+0000 7f0fbecd6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:29.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.213+0000 7f0fbecd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:09:29.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.213+0000 7f0fbecd6640 1 -- 172.21.15.18:0/881212469 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f98005080 con 0x7f0fb8143d10 2024-03-29T11:09:29.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.214+0000 7f0fb57fa640 1 -- 172.21.15.18:0/881212469 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f0fac004ae0 con 0x7f0fb8143d10 2024-03-29T11:09:29.200 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.215+0000 7f0fb57fa640 1 -- 172.21.15.18:0/881212469 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f0fac084930 con 0x7f0fb8143d10 2024-03-29T11:09:29.200 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.215+0000 7f0fbca72640 10 monclient: get_auth_request con 0x7f0fa406de30 auth_method 0 2024-03-29T11:09:29.200 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.215+0000 7f0fb57fa640 10 client.4251.objecter ms_dispatch 0x7f0fb8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:09:29.200 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.215+0000 7f0fb57fa640 3 client.4251.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:09:29.200 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.215+0000 7f0fb57fa640 3 client.4251.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:09:29.200 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.216+0000 7f0fb57fa640 20 client.4251.objecter dump_active .. 0 homeless 2024-03-29T11:09:29.200 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.216+0000 7f0fb57fa640 10 client.4251.objecter ms_handle_connect 0x7f0fa406de30 2024-03-29T11:09:29.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.224+0000 7f0fb57fa640 1 -- 172.21.15.18:0/881212469 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0fac050040 con 0x7f0fb8143d10 2024-03-29T11:09:29.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.224+0000 7f0fb57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:29.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.224+0000 7f0fb57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:09:29.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.567+0000 7f0fbecd6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:29.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.567+0000 7f0fbecd6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:29.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.567+0000 7f0fbecd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:09:29.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.567+0000 7f0fbecd6640 1 -- 172.21.15.18:0/881212469 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f98005330 con 0x7f0fb8143d10 2024-03-29T11:09:29.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.568+0000 7f0fb57fa640 1 -- 172.21.15.18:0/881212469 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f0fac052140 con 0x7f0fb8143d10 2024-03-29T11:09:29.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.568+0000 7f0fb57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:29.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.568+0000 7f0fb57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:09:29.547 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:09:29.547 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:09:29.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.571+0000 7f0fbecd6640 10 librados: watch_flush enter 2024-03-29T11:09:29.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.571+0000 7f0fbecd6640 10 librados: watch_flush exit 2024-03-29T11:09:29.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.571+0000 7f0fbecd6640 20 client.4251.objecter shutdown clearing up homeless session... 2024-03-29T11:09:29.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.571+0000 7f0fbecd6640 10 client.4251.objecter successfully canceled tick 2024-03-29T11:09:29.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.571+0000 7f0fbecd6640 1 -- 172.21.15.18:0/881212469 >> v1:172.21.15.70:6816/3101871562 conn(0x7f0fa406de30 legacy=0x7f0fa40702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:29.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.571+0000 7f0fbecd6640 10 monclient: shutdown 2024-03-29T11:09:29.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.571+0000 7f0fbecd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:29.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.571+0000 7f0fbecd6640 1 -- 172.21.15.18:0/881212469 >> v1:172.21.15.18:6789/0 conn(0x7f0fb8143d10 legacy=0x7f0fb813c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:29.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.572+0000 7f0fbecd6640 1 -- 172.21.15.18:0/881212469 shutdown_connections 2024-03-29T11:09:29.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.572+0000 7f0fbecd6640 1 -- 172.21.15.18:0/881212469 >> 172.21.15.18:0/881212469 conn(0x7f0fb8066a00 msgr2=0x7f0fb8142ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:29.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.572+0000 7f0fbecd6640 1 -- 172.21.15.18:0/881212469 shutdown_connections 2024-03-29T11:09:29.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.572+0000 7f0fbecd6640 1 -- 172.21.15.18:0/881212469 wait complete. 2024-03-29T11:09:29.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:29.572+0000 7f0fbecd6640 1 librados: shutdown 2024-03-29T11:09:29.567 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:09:36.375 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:09:36.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.542+0000 7fd1db59e640 10 monclient: get_monmap_and_config 2024-03-29T11:09:36.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.542+0000 7fd1db59e640 10 monclient: build_initial_monmap 2024-03-29T11:09:36.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.542+0000 7fd1db59e640 10 monclient: monmap: 2024-03-29T11:09:36.522 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:09:36.522 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:36.522 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:09:36.544487+0000 2024-03-29T11:09:36.522 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:09:36.544487+0000 2024-03-29T11:09:36.522 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:36.523 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:36.523 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:09:36.523 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:09:36.523 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:09:36.523 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:36.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.543+0000 7fd1db59e640 1 Processor -- start 2024-03-29T11:09:36.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.543+0000 7fd1db59e640 1 -- start start 2024-03-29T11:09:36.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.543+0000 7fd1db59e640 10 monclient: init 2024-03-29T11:09:36.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.544+0000 7fd1db59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:36.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.544+0000 7fd1db59e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:09:36.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.544+0000 7fd1db59e640 10 monclient(hunting): picked mon.noname-a con 0x7fd1d4006890 addr v1:172.21.15.18:6789/0 2024-03-29T11:09:36.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.544+0000 7fd1db59e640 10 monclient(hunting): picked mon.noname-c con 0x7fd1d4006c70 addr v1:172.21.15.70:6789/0 2024-03-29T11:09:36.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.544+0000 7fd1db59e640 10 monclient(hunting): picked mon.noname-b con 0x7fd1d402b4c0 addr v1:172.21.15.18:6790/0 2024-03-29T11:09:36.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.544+0000 7fd1db59e640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1d402c830 con 0x7fd1d4006890 2024-03-29T11:09:36.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.544+0000 7fd1db59e640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1d402afa0 con 0x7fd1d4006c70 2024-03-29T11:09:36.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.544+0000 7fd1db59e640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1d402aba0 con 0x7fd1d402b4c0 2024-03-29T11:09:36.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.544+0000 7fd1db59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:36.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.544+0000 7fd1db59e640 10 monclient(hunting): authenticate will time out at 802.817702s 2024-03-29T11:09:36.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.546+0000 7fd1d9d9b640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fd1d4006c70 0x7fd1d40c6d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.18:51916/0 (socket says 172.21.15.18:51916) 2024-03-29T11:09:36.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.546+0000 7fd1d9d9b640 1 -- 172.21.15.18:0/3071008580 learned_addr learned my addr 172.21.15.18:0/3071008580 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:09:36.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.548+0000 7fd1d959a640 1 -- 172.21.15.18:0/3071008580 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3912955160 0 0) 0x7fd1d402c830 con 0x7fd1d4006890 2024-03-29T11:09:36.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.548+0000 7fd1d959a640 10 monclient(hunting): _negotiate 2024-03-29T11:09:36.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.548+0000 7fd1d959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:36.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.548+0000 7fd1d959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:36.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.548+0000 7fd1d959a640 10 monclient(hunting): my global_id is 4254 2024-03-29T11:09:36.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.548+0000 7fd1d959a640 1 -- 172.21.15.18:0/3071008580 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1c4003520 con 0x7fd1d4006890 2024-03-29T11:09:36.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.549+0000 7fd1d959a640 1 -- 172.21.15.18:0/3071008580 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 924443070 0 0) 0x7fd1d402afa0 con 0x7fd1d4006c70 2024-03-29T11:09:36.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.549+0000 7fd1d959a640 10 monclient(hunting): _negotiate 2024-03-29T11:09:36.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.549+0000 7fd1d959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:36.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.549+0000 7fd1d959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:36.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.549+0000 7fd1d959a640 10 monclient(hunting): my global_id is 4246 2024-03-29T11:09:36.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.549+0000 7fd1d959a640 1 -- 172.21.15.18:0/3071008580 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1d402c830 con 0x7fd1d4006c70 2024-03-29T11:09:36.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.549+0000 7fd1d959a640 1 -- 172.21.15.18:0/3071008580 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1879189750 0 0) 0x7fd1c4003520 con 0x7fd1d4006890 2024-03-29T11:09:36.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.549+0000 7fd1d959a640 1 -- 172.21.15.18:0/3071008580 >> v1:172.21.15.18:6790/0 conn(0x7fd1d402b4c0 legacy=0x7fd1d40cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:09:36.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.549+0000 7fd1d959a640 1 -- 172.21.15.18:0/3071008580 >> v1:172.21.15.70:6789/0 conn(0x7fd1d4006c70 legacy=0x7fd1d40c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:36.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.549+0000 7fd1d959a640 10 monclient: _finish_hunting 0 2024-03-29T11:09:36.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.549+0000 7fd1d959a640 1 monclient: found mon.noname-a 2024-03-29T11:09:36.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.549+0000 7fd1d959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:36.532 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.549+0000 7fd1d959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-29T11:09:36.532 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.549+0000 7fd1d959a640 1 -- 172.21.15.18:0/3071008580 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d402ad30 con 0x7fd1d4006890 2024-03-29T11:09:36.532 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.549+0000 7fd1d959a640 10 monclient: _finish_auth 0 2024-03-29T11:09:36.532 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.549+0000 7fd1d959a640 10 monclient: _check_auth_tickets 2024-03-29T11:09:36.532 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.549+0000 7fd1d959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:36.532 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.549+0000 7fd1d959a640 1 -- 172.21.15.18:0/3071008580 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fd1dc04f730 con 0x7fd1d4006890 2024-03-29T11:09:36.532 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.549+0000 7fd1d959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:36.532 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.549+0000 7fd1d959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:09:36.532 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.549+0000 7fd1d959a640 10 monclient: dump: 2024-03-29T11:09:36.532 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:09:36.532 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:36.532 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:36.532 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:36.532 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:36.532 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:36.533 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:36.533 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:36.533 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:36.533 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:36.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.549+0000 7fd1d959a640 1 -- 172.21.15.18:0/3071008580 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd1dc0008d0 con 0x7fd1d4006890 2024-03-29T11:09:36.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.549+0000 7fd1d959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:36.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.549+0000 7fd1d959a640 1 -- 172.21.15.18:0/3071008580 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fd1dc0758b0 con 0x7fd1d4006890 2024-03-29T11:09:36.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.549+0000 7fd1d959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:36.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.549+0000 7fd1d959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:09:36.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.549+0000 7fd1d959a640 10 monclient: dump: 2024-03-29T11:09:36.533 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:09:36.533 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:36.533 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:36.533 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:36.533 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:36.534 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:36.534 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:36.534 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:36.534 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:36.534 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:36.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.549+0000 7fd1db59e640 5 monclient: authenticate success, global_id 4254 2024-03-29T11:09:36.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.549+0000 7fd1db59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:09:36.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.549+0000 7fd1db59e640 4 set_mon_vals no callback set 2024-03-29T11:09:36.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.550+0000 7fd1db59e640 10 monclient: shutdown 2024-03-29T11:09:36.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.550+0000 7fd1db59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:36.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.550+0000 7fd1db59e640 1 -- 172.21.15.18:0/3071008580 >> v1:172.21.15.18:6789/0 conn(0x7fd1d4006890 legacy=0x7fd1d40c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:36.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.550+0000 7fd1db59e640 1 -- 172.21.15.18:0/3071008580 shutdown_connections 2024-03-29T11:09:36.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.550+0000 7fd1db59e640 1 -- 172.21.15.18:0/3071008580 >> 172.21.15.18:0/3071008580 conn(0x7fd1d40058f0 msgr2=0x7fd1d4005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:36.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.551+0000 7fd1db59e640 1 -- 172.21.15.18:0/3071008580 shutdown_connections 2024-03-29T11:09:36.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.551+0000 7fd1db59e640 1 -- 172.21.15.18:0/3071008580 wait complete. 2024-03-29T11:09:36.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.552+0000 7fd1db59e640 10 monclient: build_initial_monmap 2024-03-29T11:09:36.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.552+0000 7fd1db59e640 10 monclient: monmap: 2024-03-29T11:09:36.536 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:09:36.536 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:09:36.536 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:09:36.536 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:09:36.536 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:36.536 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:36.536 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:09:36.536 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:09:36.537 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:09:36.537 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:36.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.552+0000 7fd1db59e640 1 librados: starting msgr at 2024-03-29T11:09:36.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.552+0000 7fd1db59e640 1 librados: starting objecter 2024-03-29T11:09:36.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.552+0000 7fd1db59e640 1 Processor -- start 2024-03-29T11:09:36.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.552+0000 7fd1db59e640 1 -- start start 2024-03-29T11:09:36.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.552+0000 7fd1db59e640 1 librados: setting wanted keys 2024-03-29T11:09:36.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.552+0000 7fd1db59e640 1 librados: calling monclient init 2024-03-29T11:09:36.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.552+0000 7fd1db59e640 10 monclient: init 2024-03-29T11:09:36.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.553+0000 7fd1db59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:36.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.553+0000 7fd1db59e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:09:36.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.553+0000 7fd1db59e640 10 monclient(hunting): picked mon.noname-c con 0x7fd1d4006890 addr v1:172.21.15.18:6790/0 2024-03-29T11:09:36.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.553+0000 7fd1db59e640 10 monclient(hunting): picked mon.noname-a con 0x7fd1d402b4c0 addr v1:172.21.15.18:6789/0 2024-03-29T11:09:36.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.553+0000 7fd1db59e640 10 monclient(hunting): picked mon.noname-b con 0x7fd1d40be630 addr v1:172.21.15.70:6789/0 2024-03-29T11:09:36.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.553+0000 7fd1db59e640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1d40c18f0 con 0x7fd1d402b4c0 2024-03-29T11:09:36.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.553+0000 7fd1db59e640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1d4152d90 con 0x7fd1d40be630 2024-03-29T11:09:36.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.553+0000 7fd1db59e640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1d40c6300 con 0x7fd1d4006890 2024-03-29T11:09:36.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.553+0000 7fd1db59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:36.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.553+0000 7fd1db59e640 10 monclient(hunting): authenticate will time out at 802.826582s 2024-03-29T11:09:36.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.553+0000 7fd1dad9d640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fd1d40be630 0x7fd1d41680d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.18:52368/0 (socket says 172.21.15.18:52368) 2024-03-29T11:09:36.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.553+0000 7fd1dad9d640 1 -- 172.21.15.18:0/2231375632 learned_addr learned my addr 172.21.15.18:0/2231375632 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:09:36.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.554+0000 7fd1caffd640 10 client.?.objecter ms_handle_connect 0x7fd1d40be630 2024-03-29T11:09:36.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.554+0000 7fd1caffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:36.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.556+0000 7fd1caffd640 10 client.?.objecter ms_handle_connect 0x7fd1d4006890 2024-03-29T11:09:36.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.556+0000 7fd1caffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:36.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.556+0000 7fd1caffd640 1 -- 172.21.15.18:0/2231375632 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1302616619 0 0) 0x7fd1d40c6300 con 0x7fd1d4006890 2024-03-29T11:09:36.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.556+0000 7fd1caffd640 10 monclient(hunting): _negotiate 2024-03-29T11:09:36.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.556+0000 7fd1caffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:36.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.556+0000 7fd1caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:36.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.556+0000 7fd1caffd640 10 monclient(hunting): my global_id is 4256 2024-03-29T11:09:36.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.556+0000 7fd1caffd640 1 -- 172.21.15.18:0/2231375632 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1bc004eb0 con 0x7fd1d4006890 2024-03-29T11:09:36.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.556+0000 7fd1caffd640 1 -- 172.21.15.18:0/2231375632 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3020138883 0 0) 0x7fd1bc004eb0 con 0x7fd1d4006890 2024-03-29T11:09:36.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.556+0000 7fd1caffd640 1 -- 172.21.15.18:0/2231375632 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1d40c6300 con 0x7fd1d4006890 2024-03-29T11:09:36.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.556+0000 7fd1caffd640 1 -- 172.21.15.18:0/2231375632 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fd1dc04f400 con 0x7fd1d4006890 2024-03-29T11:09:36.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.556+0000 7fd1caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:36.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.556+0000 7fd1caffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:09:36.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.556+0000 7fd1caffd640 10 monclient(hunting): dump: 2024-03-29T11:09:36.539 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:09:36.539 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:36.539 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:36.539 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:36.540 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:36.540 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:36.540 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:36.540 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:36.540 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:36.540 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:36.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.556+0000 7fd1caffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:09:36.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.559+0000 7fd1caffd640 1 -- 172.21.15.18:0/2231375632 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1801705793 0 0) 0x7fd1d40c6300 con 0x7fd1d4006890 2024-03-29T11:09:36.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.559+0000 7fd1caffd640 1 -- 172.21.15.18:0/2231375632 >> v1:172.21.15.70:6789/0 conn(0x7fd1d40be630 legacy=0x7fd1d41680d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:36.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.559+0000 7fd1caffd640 1 -- 172.21.15.18:0/2231375632 >> v1:172.21.15.18:6789/0 conn(0x7fd1d402b4c0 legacy=0x7fd1d40bdf20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:09:36.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.559+0000 7fd1caffd640 10 monclient: _finish_hunting 0 2024-03-29T11:09:36.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.559+0000 7fd1caffd640 1 monclient: found mon.c 2024-03-29T11:09:36.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.560+0000 7fd1caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:36.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.560+0000 7fd1caffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:09:36.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.560+0000 7fd1caffd640 1 -- 172.21.15.18:0/2231375632 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1d40c4790 con 0x7fd1d4006890 2024-03-29T11:09:36.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.560+0000 7fd1caffd640 10 monclient: _finish_auth 0 2024-03-29T11:09:36.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.560+0000 7fd1caffd640 10 monclient: _check_auth_tickets 2024-03-29T11:09:36.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.560+0000 7fd1caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:36.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.560+0000 7fd1db59e640 5 monclient: authenticate success, global_id 4256 2024-03-29T11:09:36.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.560+0000 7fd1db59e640 10 monclient: _renew_subs 2024-03-29T11:09:36.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.560+0000 7fd1db59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:09:36.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.560+0000 7fd1db59e640 1 -- 172.21.15.18:0/2231375632 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1d414ea80 con 0x7fd1d4006890 2024-03-29T11:09:36.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.560+0000 7fd1db59e640 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:09:36.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.560+0000 7fd1db59e640 10 monclient: _renew_subs 2024-03-29T11:09:36.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.560+0000 7fd1db59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:09:36.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.560+0000 7fd1db59e640 1 -- 172.21.15.18:0/2231375632 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1d4152990 con 0x7fd1d4006890 2024-03-29T11:09:36.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.560+0000 7fd1db59e640 1 librados: init done 2024-03-29T11:09:36.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.561+0000 7fd1caffd640 1 -- 172.21.15.18:0/2231375632 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd1dc04fdd0 con 0x7fd1d4006890 2024-03-29T11:09:36.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.561+0000 7fd1caffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:36.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.561+0000 7fd1caffd640 1 -- 172.21.15.18:0/2231375632 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fd1dc067bf0 con 0x7fd1d4006890 2024-03-29T11:09:36.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.561+0000 7fd1caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:36.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.561+0000 7fd1caffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:09:36.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.561+0000 7fd1caffd640 10 monclient: dump: 2024-03-29T11:09:36.542 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:09:36.542 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:36.543 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:36.543 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:36.543 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:36.543 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:36.543 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:36.543 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:36.543 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:36.543 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:36.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.561+0000 7fd1d8d99640 4 set_mon_vals no callback set 2024-03-29T11:09:36.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.561+0000 7fd1db59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:36.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.562+0000 7fd1db59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:36.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.562+0000 7fd1db59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:09:36.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.562+0000 7fd1db59e640 1 -- 172.21.15.18:0/2231375632 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1c0005080 con 0x7fd1d4006890 2024-03-29T11:09:36.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.564+0000 7fd1caffd640 1 -- 172.21.15.18:0/2231375632 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7fd1dc067e10 con 0x7fd1d4006890 2024-03-29T11:09:36.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.564+0000 7fd1caffd640 1 -- 172.21.15.18:0/2231375632 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7fd1dc0ce170 con 0x7fd1d4006890 2024-03-29T11:09:36.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.564+0000 7fd1caffd640 10 client.4256.objecter ms_dispatch 0x7fd1dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:09:36.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.564+0000 7fd1caffd640 3 client.4256.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:09:36.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.564+0000 7fd1caffd640 3 client.4256.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:09:36.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.564+0000 7fd1caffd640 20 client.4256.objecter dump_active .. 0 homeless 2024-03-29T11:09:36.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.564+0000 7fd1d9d9b640 10 monclient: get_auth_request con 0x7fd1bc06da80 auth_method 0 2024-03-29T11:09:36.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.565+0000 7fd1caffd640 10 client.4256.objecter ms_handle_connect 0x7fd1bc06da80 2024-03-29T11:09:36.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.573+0000 7fd1caffd640 1 -- 172.21.15.18:0/2231375632 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd1dc0ca410 con 0x7fd1d4006890 2024-03-29T11:09:36.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.573+0000 7fd1caffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:36.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.573+0000 7fd1caffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:09:36.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.909+0000 7fd1db59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:36.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.909+0000 7fd1db59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:36.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.909+0000 7fd1db59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:09:36.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.909+0000 7fd1db59e640 1 -- 172.21.15.18:0/2231375632 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd1c0005330 con 0x7fd1d4006890 2024-03-29T11:09:36.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.911+0000 7fd1caffd640 1 -- 172.21.15.18:0/2231375632 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7fd1dc0ca5b0 con 0x7fd1d4006890 2024-03-29T11:09:36.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.911+0000 7fd1caffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:36.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.911+0000 7fd1caffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:09:36.891 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:09:36.891 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:09:36.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.915+0000 7fd1db59e640 10 librados: watch_flush enter 2024-03-29T11:09:36.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.915+0000 7fd1db59e640 10 librados: watch_flush exit 2024-03-29T11:09:36.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.915+0000 7fd1db59e640 20 client.4256.objecter shutdown clearing up homeless session... 2024-03-29T11:09:36.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.915+0000 7fd1db59e640 10 client.4256.objecter successfully canceled tick 2024-03-29T11:09:36.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.915+0000 7fd1db59e640 1 -- 172.21.15.18:0/2231375632 >> v1:172.21.15.70:6816/3101871562 conn(0x7fd1bc06da80 legacy=0x7fd1bc06ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:36.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.915+0000 7fd1db59e640 10 monclient: shutdown 2024-03-29T11:09:36.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.915+0000 7fd1db59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:36.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.915+0000 7fd1db59e640 1 -- 172.21.15.18:0/2231375632 >> v1:172.21.15.18:6790/0 conn(0x7fd1d4006890 legacy=0x7fd1d40c5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:36.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.915+0000 7fd1db59e640 1 -- 172.21.15.18:0/2231375632 shutdown_connections 2024-03-29T11:09:36.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.915+0000 7fd1db59e640 1 -- 172.21.15.18:0/2231375632 >> 172.21.15.18:0/2231375632 conn(0x7fd1d40058f0 msgr2=0x7fd1d4038130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:36.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.915+0000 7fd1db59e640 1 -- 172.21.15.18:0/2231375632 shutdown_connections 2024-03-29T11:09:36.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.915+0000 7fd1db59e640 1 -- 172.21.15.18:0/2231375632 wait complete. 2024-03-29T11:09:36.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:36.915+0000 7fd1db59e640 1 librados: shutdown 2024-03-29T11:09:36.911 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:09:43.719 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:09:43.872 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.894+0000 7f2007d2b640 10 monclient: get_monmap_and_config 2024-03-29T11:09:43.872 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.894+0000 7f2007d2b640 10 monclient: build_initial_monmap 2024-03-29T11:09:43.873 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.894+0000 7f2007d2b640 10 monclient: monmap: 2024-03-29T11:09:43.873 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:09:43.873 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:43.873 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:09:43.895117+0000 2024-03-29T11:09:43.873 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:09:43.895117+0000 2024-03-29T11:09:43.873 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:43.873 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:43.873 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:09:43.873 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:09:43.873 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:09:43.873 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:43.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.897+0000 7f2007d2b640 1 Processor -- start 2024-03-29T11:09:43.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.900+0000 7f2007d2b640 1 -- start start 2024-03-29T11:09:43.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.900+0000 7f2007d2b640 10 monclient: init 2024-03-29T11:09:43.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.903+0000 7f2007d2b640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:43.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.903+0000 7f2007d2b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:09:43.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.903+0000 7f2007d2b640 10 monclient(hunting): picked mon.noname-a con 0x7f20001422b0 addr v1:172.21.15.18:6789/0 2024-03-29T11:09:43.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.903+0000 7f2007d2b640 10 monclient(hunting): picked mon.noname-b con 0x7f2000142690 addr v1:172.21.15.18:6790/0 2024-03-29T11:09:43.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.905+0000 7f2007d2b640 10 monclient(hunting): picked mon.noname-c con 0x7f20000b17d0 addr v1:172.21.15.70:6789/0 2024-03-29T11:09:43.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.905+0000 7f2007d2b640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2000142c80 con 0x7f20001422b0 2024-03-29T11:09:43.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.905+0000 7f2007d2b640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20000aae40 con 0x7f20000b17d0 2024-03-29T11:09:43.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.905+0000 7f2007d2b640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f200013fbd0 con 0x7f2000142690 2024-03-29T11:09:43.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.905+0000 7f2007d2b640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:43.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.905+0000 7f2007d2b640 10 monclient(hunting): authenticate will time out at 810.177316s 2024-03-29T11:09:43.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.908+0000 7f20062c8640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f20000b17d0 0x7f20000b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.18:60149/0 (socket says 172.21.15.18:60149) 2024-03-29T11:09:43.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.909+0000 7f20062c8640 1 -- 172.21.15.18:0/1817270313 learned_addr learned my addr 172.21.15.18:0/1817270313 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:09:43.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.910+0000 7f2004ac5640 1 -- 172.21.15.18:0/1817270313 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2003350813 0 0) 0x7f200013fbd0 con 0x7f2000142690 2024-03-29T11:09:43.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.910+0000 7f2004ac5640 10 monclient(hunting): _negotiate 2024-03-29T11:09:43.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.910+0000 7f2004ac5640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:43.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.910+0000 7f2004ac5640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:43.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.910+0000 7f2004ac5640 10 monclient(hunting): my global_id is 4259 2024-03-29T11:09:43.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.910+0000 7f2004ac5640 1 -- 172.21.15.18:0/1817270313 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fe8003520 con 0x7f2000142690 2024-03-29T11:09:43.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.914+0000 7f2004ac5640 1 -- 172.21.15.18:0/1817270313 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2114177588 0 0) 0x7f20000aae40 con 0x7f20000b17d0 2024-03-29T11:09:43.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.914+0000 7f2004ac5640 10 monclient(hunting): _negotiate 2024-03-29T11:09:43.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.914+0000 7f2004ac5640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:43.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.914+0000 7f2004ac5640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:43.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.914+0000 7f2004ac5640 10 monclient(hunting): my global_id is 4252 2024-03-29T11:09:43.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.914+0000 7f2004ac5640 1 -- 172.21.15.18:0/1817270313 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f200013fbd0 con 0x7f20000b17d0 2024-03-29T11:09:43.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.914+0000 7f2004ac5640 1 -- 172.21.15.18:0/1817270313 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3340059789 0 0) 0x7f1fe8003520 con 0x7f2000142690 2024-03-29T11:09:43.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.914+0000 7f2004ac5640 1 -- 172.21.15.18:0/1817270313 >> v1:172.21.15.70:6789/0 conn(0x7f20000b17d0 legacy=0x7f20000b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:43.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.914+0000 7f2004ac5640 1 -- 172.21.15.18:0/1817270313 >> v1:172.21.15.18:6789/0 conn(0x7f20001422b0 legacy=0x7f200013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:09:43.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.914+0000 7f2004ac5640 10 monclient: _finish_hunting 0 2024-03-29T11:09:43.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.914+0000 7f2004ac5640 1 monclient: found mon.noname-b 2024-03-29T11:09:43.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.914+0000 7f2004ac5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:43.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.914+0000 7f2004ac5640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:09:43.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.914+0000 7f2004ac5640 1 -- 172.21.15.18:0/1817270313 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2000151940 con 0x7f2000142690 2024-03-29T11:09:43.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.914+0000 7f2004ac5640 10 monclient: _finish_auth 0 2024-03-29T11:09:43.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.914+0000 7f2004ac5640 10 monclient: _check_auth_tickets 2024-03-29T11:09:43.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.914+0000 7f2004ac5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:43.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.914+0000 7f2004ac5640 1 -- 172.21.15.18:0/1817270313 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f1ff4001690 con 0x7f2000142690 2024-03-29T11:09:43.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.914+0000 7f2004ac5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:43.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.914+0000 7f2004ac5640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:09:43.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.914+0000 7f2004ac5640 10 monclient: dump: 2024-03-29T11:09:43.894 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:09:43.894 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:43.894 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:43.894 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:43.894 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:43.894 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:43.894 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:43.894 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:43.894 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:43.894 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:43.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.914+0000 7f2007d2b640 5 monclient: authenticate success, global_id 4259 2024-03-29T11:09:43.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.914+0000 7f2007d2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:43.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.914+0000 7f2004ac5640 1 -- 172.21.15.18:0/1817270313 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ff4001870 con 0x7f2000142690 2024-03-29T11:09:43.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.914+0000 7f2004ac5640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:43.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.915+0000 7f2007d2b640 10 monclient: get_monmap_and_config success 2024-03-29T11:09:43.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.915+0000 7f2007d2b640 4 set_mon_vals no callback set 2024-03-29T11:09:43.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.915+0000 7f2004ac5640 1 -- 172.21.15.18:0/1817270313 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f1ff4001c00 con 0x7f2000142690 2024-03-29T11:09:43.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.915+0000 7f2004ac5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:43.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.915+0000 7f2004ac5640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:09:43.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.915+0000 7f2004ac5640 10 monclient: dump: 2024-03-29T11:09:43.899 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:09:43.899 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:43.899 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:43.899 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:43.899 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:43.899 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:43.899 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:43.899 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:43.899 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:43.899 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:43.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.915+0000 7f2007d2b640 10 monclient: shutdown 2024-03-29T11:09:43.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.915+0000 7f2007d2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:43.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.915+0000 7f2007d2b640 1 -- 172.21.15.18:0/1817270313 >> v1:172.21.15.18:6790/0 conn(0x7f2000142690 legacy=0x7f20000b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:43.900 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.916+0000 7f2007d2b640 1 -- 172.21.15.18:0/1817270313 shutdown_connections 2024-03-29T11:09:43.900 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.916+0000 7f2007d2b640 1 -- 172.21.15.18:0/1817270313 >> 172.21.15.18:0/1817270313 conn(0x7f2000066a00 msgr2=0x7f2000066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:43.900 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.916+0000 7f2007d2b640 1 -- 172.21.15.18:0/1817270313 shutdown_connections 2024-03-29T11:09:43.900 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.916+0000 7f2007d2b640 1 -- 172.21.15.18:0/1817270313 wait complete. 2024-03-29T11:09:43.900 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.917+0000 7f2007d2b640 10 monclient: build_initial_monmap 2024-03-29T11:09:43.900 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.917+0000 7f2007d2b640 10 monclient: monmap: 2024-03-29T11:09:43.900 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:09:43.900 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:09:43.900 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:09:43.900 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:09:43.900 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:43.900 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:43.900 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:09:43.901 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:09:43.901 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:09:43.901 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:43.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.917+0000 7f2007d2b640 1 librados: starting msgr at 2024-03-29T11:09:43.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.917+0000 7f2007d2b640 1 librados: starting objecter 2024-03-29T11:09:43.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.917+0000 7f2007d2b640 1 Processor -- start 2024-03-29T11:09:43.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.917+0000 7f2007d2b640 1 -- start start 2024-03-29T11:09:43.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.917+0000 7f2007d2b640 1 librados: setting wanted keys 2024-03-29T11:09:43.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.917+0000 7f2007d2b640 1 librados: calling monclient init 2024-03-29T11:09:43.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.917+0000 7f2007d2b640 10 monclient: init 2024-03-29T11:09:43.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.918+0000 7f2007d2b640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:43.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.918+0000 7f2007d2b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:09:43.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.918+0000 7f2007d2b640 10 monclient(hunting): picked mon.noname-b con 0x7f20000b17d0 addr v1:172.21.15.70:6789/0 2024-03-29T11:09:43.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.918+0000 7f2007d2b640 10 monclient(hunting): picked mon.noname-a con 0x7f20001422b0 addr v1:172.21.15.18:6789/0 2024-03-29T11:09:43.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.922+0000 7f2007d2b640 10 monclient(hunting): picked mon.noname-c con 0x7f2000142690 addr v1:172.21.15.18:6790/0 2024-03-29T11:09:43.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.922+0000 7f2007d2b640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20001312c0 con 0x7f20001422b0 2024-03-29T11:09:43.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.922+0000 7f2007d2b640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20000b1440 con 0x7f20000b17d0 2024-03-29T11:09:43.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.922+0000 7f2007d2b640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20000acf90 con 0x7f2000142690 2024-03-29T11:09:43.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.922+0000 7f2007d2b640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:43.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.922+0000 7f2007d2b640 10 monclient(hunting): authenticate will time out at 810.194309s 2024-03-29T11:09:43.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.925+0000 7f20062c8640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f2000142690 0x7f20001e20e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:33101/0 (socket says 172.21.15.18:33101) 2024-03-29T11:09:43.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.925+0000 7f20062c8640 1 -- 172.21.15.18:0/3943213669 learned_addr learned my addr 172.21.15.18:0/3943213669 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:09:43.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.925+0000 7f1ffe7fc640 10 client.?.objecter ms_handle_connect 0x7f2000142690 2024-03-29T11:09:43.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.925+0000 7f1ffe7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:43.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.926+0000 7f1ffe7fc640 1 -- 172.21.15.18:0/3943213669 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3115005433 0 0) 0x7f20000acf90 con 0x7f2000142690 2024-03-29T11:09:43.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.926+0000 7f1ffe7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:09:43.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.926+0000 7f1ffe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:43.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.926+0000 7f1ffe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:43.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.926+0000 7f1ffe7fc640 10 monclient(hunting): my global_id is 4262 2024-03-29T11:09:43.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.926+0000 7f1ffe7fc640 1 -- 172.21.15.18:0/3943213669 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1fec004eb0 con 0x7f2000142690 2024-03-29T11:09:43.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.927+0000 7f1ffe7fc640 1 -- 172.21.15.18:0/3943213669 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2879969124 0 0) 0x7f1fec004eb0 con 0x7f2000142690 2024-03-29T11:09:43.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.927+0000 7f1ffe7fc640 1 -- 172.21.15.18:0/3943213669 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1fec005300 con 0x7f2000142690 2024-03-29T11:09:43.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.927+0000 7f1ffe7fc640 1 -- 172.21.15.18:0/3943213669 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f1ff0001cb0 con 0x7f2000142690 2024-03-29T11:09:43.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.927+0000 7f1ffe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:43.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.927+0000 7f1ffe7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:09:43.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.927+0000 7f1ffe7fc640 10 monclient(hunting): dump: 2024-03-29T11:09:43.911 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:09:43.911 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:43.911 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:43.911 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:43.911 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:43.911 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:43.911 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:43.912 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:43.912 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:43.912 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:43.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.927+0000 7f1ffe7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:09:43.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.927+0000 7f1ffe7fc640 1 -- 172.21.15.18:0/3943213669 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 909697615 0 0) 0x7f1fec005300 con 0x7f2000142690 2024-03-29T11:09:43.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.927+0000 7f1ffe7fc640 1 -- 172.21.15.18:0/3943213669 >> v1:172.21.15.70:6789/0 conn(0x7f20000b17d0 legacy=0x7f20001dae10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:09:43.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.929+0000 7f1ffe7fc640 1 -- 172.21.15.18:0/3943213669 >> v1:172.21.15.18:6789/0 conn(0x7f20001422b0 legacy=0x7f200012fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:43.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.929+0000 7f1ffe7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:09:43.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.929+0000 7f1ffe7fc640 1 monclient: found mon.c 2024-03-29T11:09:43.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.929+0000 7f1ffe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:43.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.929+0000 7f1ffe7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:09:43.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.929+0000 7f1ffe7fc640 1 -- 172.21.15.18:0/3943213669 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20000ad200 con 0x7f2000142690 2024-03-29T11:09:43.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.929+0000 7f1ffe7fc640 10 monclient: _finish_auth 0 2024-03-29T11:09:43.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.929+0000 7f1ffe7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:09:43.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.929+0000 7f1ffe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:43.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.929+0000 7f1ffe7fc640 10 client.?.objecter ms_handle_connect 0x7f20001422b0 2024-03-29T11:09:43.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.929+0000 7f1ffe7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:43.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.929+0000 7f2007d2b640 5 monclient: authenticate success, global_id 4262 2024-03-29T11:09:43.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.929+0000 7f2007d2b640 10 monclient: _renew_subs 2024-03-29T11:09:43.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.929+0000 7f2007d2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:09:43.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.929+0000 7f2007d2b640 1 -- 172.21.15.18:0/3943213669 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20000ad4e0 con 0x7f2000142690 2024-03-29T11:09:43.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.929+0000 7f2007d2b640 10 client.4262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:09:43.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.929+0000 7f2007d2b640 10 monclient: _renew_subs 2024-03-29T11:09:43.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.929+0000 7f2007d2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:09:43.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.929+0000 7f2007d2b640 1 -- 172.21.15.18:0/3943213669 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f20001db700 con 0x7f2000142690 2024-03-29T11:09:43.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.929+0000 7f2007d2b640 1 librados: init done 2024-03-29T11:09:43.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.931+0000 7f1ffe7fc640 1 -- 172.21.15.18:0/3943213669 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ff0004b60 con 0x7f2000142690 2024-03-29T11:09:43.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.931+0000 7f1ffe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:43.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.931+0000 7f1ffe7fc640 1 -- 172.21.15.18:0/3943213669 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f1ff0001cb0 con 0x7f2000142690 2024-03-29T11:09:43.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.931+0000 7f1ffe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:43.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.931+0000 7f1ffe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:09:43.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.931+0000 7f1ffe7fc640 10 monclient: dump: 2024-03-29T11:09:43.914 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:09:43.914 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:43.914 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:43.914 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:43.914 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:43.915 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:43.915 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:43.915 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:43.915 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:43.915 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:43.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.931+0000 7f1fe7fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:43.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.931+0000 7f1fe7fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:43.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.931+0000 7f1fe7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:09:43.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.931+0000 7f1fe7fff640 1 -- 172.21.15.18:0/3943213669 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fe0005080 con 0x7f2000142690 2024-03-29T11:09:43.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.932+0000 7f1ffe7fc640 1 -- 172.21.15.18:0/3943213669 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f1ff001abc0 con 0x7f2000142690 2024-03-29T11:09:43.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.932+0000 7f1ffffff640 4 set_mon_vals no callback set 2024-03-29T11:09:43.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.932+0000 7f1ffe7fc640 1 -- 172.21.15.18:0/3943213669 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f1ff0054780 con 0x7f2000142690 2024-03-29T11:09:43.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.932+0000 7f1ffe7fc640 10 client.4262.objecter ms_dispatch 0x7f2000000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:09:43.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.932+0000 7f1ffe7fc640 3 client.4262.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:09:43.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.932+0000 7f1ffe7fc640 3 client.4262.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:09:43.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.933+0000 7f1ffe7fc640 20 client.4262.objecter dump_active .. 0 homeless 2024-03-29T11:09:43.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.933+0000 7f2005ac7640 10 monclient: get_auth_request con 0x7f1fec06e220 auth_method 0 2024-03-29T11:09:43.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.933+0000 7f1ffe7fc640 10 client.4262.objecter ms_handle_connect 0x7f1fec06e220 2024-03-29T11:09:43.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.941+0000 7f1ffe7fc640 1 -- 172.21.15.18:0/3943213669 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1ff0050a20 con 0x7f2000142690 2024-03-29T11:09:43.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.941+0000 7f1ffe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:43.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:43.941+0000 7f1ffe7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:09:44.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:44.269+0000 7f1fe7fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:44.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:44.269+0000 7f1fe7fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:44.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:44.269+0000 7f1fe7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:09:44.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:44.269+0000 7f1fe7fff640 1 -- 172.21.15.18:0/3943213669 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1fe0005330 con 0x7f2000142690 2024-03-29T11:09:44.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:44.270+0000 7f1ffe7fc640 1 -- 172.21.15.18:0/3943213669 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f1ff0050bc0 con 0x7f2000142690 2024-03-29T11:09:44.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:44.270+0000 7f1ffe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:44.249 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:09:44.249 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:09:44.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:44.270+0000 7f1ffe7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:09:44.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:44.273+0000 7f1fe7fff640 10 librados: watch_flush enter 2024-03-29T11:09:44.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:44.273+0000 7f1fe7fff640 10 librados: watch_flush exit 2024-03-29T11:09:44.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:44.273+0000 7f1fe7fff640 20 client.4262.objecter shutdown clearing up homeless session... 2024-03-29T11:09:44.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:44.273+0000 7f1fe7fff640 10 client.4262.objecter successfully canceled tick 2024-03-29T11:09:44.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:44.273+0000 7f1fe7fff640 1 -- 172.21.15.18:0/3943213669 >> v1:172.21.15.70:6816/3101871562 conn(0x7f1fec06e220 legacy=0x7f1fec0706e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:44.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:44.273+0000 7f1fe7fff640 10 monclient: shutdown 2024-03-29T11:09:44.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:44.273+0000 7f1fe7fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:44.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:44.273+0000 7f1fe7fff640 1 -- 172.21.15.18:0/3943213669 >> v1:172.21.15.18:6790/0 conn(0x7f2000142690 legacy=0x7f20001e20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:44.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:44.273+0000 7f1fe7fff640 1 -- 172.21.15.18:0/3943213669 shutdown_connections 2024-03-29T11:09:44.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:44.273+0000 7f1fe7fff640 1 -- 172.21.15.18:0/3943213669 >> 172.21.15.18:0/3943213669 conn(0x7f2000066a00 msgr2=0x7f200013d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:44.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:44.274+0000 7f1fe7fff640 1 -- 172.21.15.18:0/3943213669 shutdown_connections 2024-03-29T11:09:44.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:44.274+0000 7f1fe7fff640 1 -- 172.21.15.18:0/3943213669 wait complete. 2024-03-29T11:09:44.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:44.274+0000 7f1fe7fff640 1 librados: shutdown 2024-03-29T11:09:44.272 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:09:51.079 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:09:51.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.259+0000 7f6b8f1c7640 10 monclient: get_monmap_and_config 2024-03-29T11:09:51.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.259+0000 7f6b8f1c7640 10 monclient: build_initial_monmap 2024-03-29T11:09:51.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.259+0000 7f6b8f1c7640 10 monclient: monmap: 2024-03-29T11:09:51.239 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:09:51.239 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:51.239 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:09:51.260555+0000 2024-03-29T11:09:51.239 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:09:51.260555+0000 2024-03-29T11:09:51.240 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:51.240 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:51.240 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:09:51.240 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:09:51.240 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:09:51.240 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:51.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.261+0000 7f6b8f1c7640 1 Processor -- start 2024-03-29T11:09:51.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.261+0000 7f6b8f1c7640 1 -- start start 2024-03-29T11:09:51.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.261+0000 7f6b8f1c7640 10 monclient: init 2024-03-29T11:09:51.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.262+0000 7f6b8f1c7640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:51.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.262+0000 7f6b8f1c7640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:09:51.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.262+0000 7f6b8f1c7640 10 monclient(hunting): picked mon.noname-b con 0x7f6b880a9930 addr v1:172.21.15.18:6790/0 2024-03-29T11:09:51.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.262+0000 7f6b8f1c7640 10 monclient(hunting): picked mon.noname-a con 0x7f6b88149dc0 addr v1:172.21.15.18:6789/0 2024-03-29T11:09:51.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.262+0000 7f6b8f1c7640 10 monclient(hunting): picked mon.noname-c con 0x7f6b8814ff40 addr v1:172.21.15.70:6789/0 2024-03-29T11:09:51.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.262+0000 7f6b8f1c7640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b880aae60 con 0x7f6b88149dc0 2024-03-29T11:09:51.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.262+0000 7f6b8f1c7640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b880b4310 con 0x7f6b8814ff40 2024-03-29T11:09:51.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.262+0000 7f6b8f1c7640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b880afdd0 con 0x7f6b880a9930 2024-03-29T11:09:51.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.262+0000 7f6b8f1c7640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:51.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.262+0000 7f6b8f1c7640 10 monclient(hunting): authenticate will time out at 817.534270s 2024-03-29T11:09:51.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.267+0000 7f6b8d764640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f6b8814ff40 0x7f6b88152330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.18:54383/0 (socket says 172.21.15.18:54383) 2024-03-29T11:09:51.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.267+0000 7f6b8d764640 1 -- 172.21.15.18:0/2675944718 learned_addr learned my addr 172.21.15.18:0/2675944718 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:09:51.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.268+0000 7f6b877fe640 1 -- 172.21.15.18:0/2675944718 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2665916952 0 0) 0x7f6b880b4310 con 0x7f6b8814ff40 2024-03-29T11:09:51.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.268+0000 7f6b877fe640 10 monclient(hunting): _negotiate 2024-03-29T11:09:51.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.268+0000 7f6b877fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:51.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.268+0000 7f6b877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:51.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.268+0000 7f6b877fe640 10 monclient(hunting): my global_id is 4255 2024-03-29T11:09:51.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.268+0000 7f6b877fe640 1 -- 172.21.15.18:0/2675944718 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b70003520 con 0x7f6b8814ff40 2024-03-29T11:09:51.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.269+0000 7f6b877fe640 1 -- 172.21.15.18:0/2675944718 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2228248737 0 0) 0x7f6b880aae60 con 0x7f6b88149dc0 2024-03-29T11:09:51.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.269+0000 7f6b877fe640 10 monclient(hunting): _negotiate 2024-03-29T11:09:51.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.269+0000 7f6b877fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:51.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.269+0000 7f6b877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:51.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.269+0000 7f6b877fe640 10 monclient(hunting): my global_id is 4260 2024-03-29T11:09:51.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.269+0000 7f6b877fe640 1 -- 172.21.15.18:0/2675944718 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b880b4310 con 0x7f6b88149dc0 2024-03-29T11:09:51.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.270+0000 7f6b877fe640 1 -- 172.21.15.18:0/2675944718 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 225504957 0 0) 0x7f6b880b4310 con 0x7f6b88149dc0 2024-03-29T11:09:51.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.270+0000 7f6b877fe640 1 -- 172.21.15.18:0/2675944718 >> v1:172.21.15.18:6790/0 conn(0x7f6b880a9930 legacy=0x7f6b88143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:51.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.270+0000 7f6b877fe640 1 -- 172.21.15.18:0/2675944718 >> v1:172.21.15.70:6789/0 conn(0x7f6b8814ff40 legacy=0x7f6b88152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:51.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.270+0000 7f6b877fe640 10 monclient: _finish_hunting 0 2024-03-29T11:09:51.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.270+0000 7f6b877fe640 1 monclient: found mon.noname-a 2024-03-29T11:09:51.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.270+0000 7f6b877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:51.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.270+0000 7f6b877fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-29T11:09:51.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.270+0000 7f6b877fe640 1 -- 172.21.15.18:0/2675944718 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b8814a950 con 0x7f6b88149dc0 2024-03-29T11:09:51.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.271+0000 7f6b877fe640 10 monclient: _finish_auth 0 2024-03-29T11:09:51.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.271+0000 7f6b877fe640 10 monclient: _check_auth_tickets 2024-03-29T11:09:51.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.271+0000 7f6b877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:51.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.271+0000 7f6b877fe640 1 -- 172.21.15.18:0/2675944718 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f6b7c001670 con 0x7f6b88149dc0 2024-03-29T11:09:51.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.271+0000 7f6b877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:51.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.271+0000 7f6b877fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:09:51.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.271+0000 7f6b877fe640 10 monclient: dump: 2024-03-29T11:09:51.250 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:09:51.250 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:51.250 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:51.250 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:51.250 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:51.250 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:51.250 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:51.250 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:51.250 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:51.251 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:51.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.271+0000 7f6b8f1c7640 5 monclient: authenticate success, global_id 4260 2024-03-29T11:09:51.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.271+0000 7f6b8f1c7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:51.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.271+0000 7f6b877fe640 1 -- 172.21.15.18:0/2675944718 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b7c0017d0 con 0x7f6b88149dc0 2024-03-29T11:09:51.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.271+0000 7f6b877fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:51.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.271+0000 7f6b877fe640 1 -- 172.21.15.18:0/2675944718 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f6b7c001ba0 con 0x7f6b88149dc0 2024-03-29T11:09:51.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.271+0000 7f6b877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:51.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.271+0000 7f6b877fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:09:51.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.271+0000 7f6b877fe640 10 monclient: dump: 2024-03-29T11:09:51.252 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:09:51.252 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:51.252 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:51.252 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:51.252 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:51.252 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:51.252 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:51.252 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:51.252 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:51.252 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:51.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.271+0000 7f6b8f1c7640 10 monclient: get_monmap_and_config success 2024-03-29T11:09:51.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.271+0000 7f6b8f1c7640 4 set_mon_vals no callback set 2024-03-29T11:09:51.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.272+0000 7f6b8f1c7640 10 monclient: shutdown 2024-03-29T11:09:51.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.272+0000 7f6b8f1c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:51.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.272+0000 7f6b8f1c7640 1 -- 172.21.15.18:0/2675944718 >> v1:172.21.15.18:6789/0 conn(0x7f6b88149dc0 legacy=0x7f6b8814a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:51.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.272+0000 7f6b8f1c7640 1 -- 172.21.15.18:0/2675944718 shutdown_connections 2024-03-29T11:09:51.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.272+0000 7f6b8f1c7640 1 -- 172.21.15.18:0/2675944718 >> 172.21.15.18:0/2675944718 conn(0x7f6b880b35e0 msgr2=0x7f6b880b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:51.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.272+0000 7f6b8f1c7640 1 -- 172.21.15.18:0/2675944718 shutdown_connections 2024-03-29T11:09:51.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.273+0000 7f6b8f1c7640 1 -- 172.21.15.18:0/2675944718 wait complete. 2024-03-29T11:09:51.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.274+0000 7f6b8f1c7640 10 monclient: build_initial_monmap 2024-03-29T11:09:51.259 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.274+0000 7f6b8f1c7640 10 monclient: monmap: 2024-03-29T11:09:51.259 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:09:51.259 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:09:51.259 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:09:51.259 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:09:51.259 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:51.259 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:51.259 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:09:51.259 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:09:51.259 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:09:51.259 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:51.259 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.274+0000 7f6b8f1c7640 1 librados: starting msgr at 2024-03-29T11:09:51.259 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.274+0000 7f6b8f1c7640 1 librados: starting objecter 2024-03-29T11:09:51.259 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.275+0000 7f6b8f1c7640 1 Processor -- start 2024-03-29T11:09:51.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.275+0000 7f6b8f1c7640 1 -- start start 2024-03-29T11:09:51.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.275+0000 7f6b8f1c7640 1 librados: setting wanted keys 2024-03-29T11:09:51.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.275+0000 7f6b8f1c7640 1 librados: calling monclient init 2024-03-29T11:09:51.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.275+0000 7f6b8f1c7640 10 monclient: init 2024-03-29T11:09:51.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.276+0000 7f6b8f1c7640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:51.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.276+0000 7f6b8f1c7640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:09:51.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.276+0000 7f6b8f1c7640 10 monclient(hunting): picked mon.noname-c con 0x7f6b880a9930 addr v1:172.21.15.18:6790/0 2024-03-29T11:09:51.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.276+0000 7f6b8f1c7640 10 monclient(hunting): picked mon.noname-a con 0x7f6b88149dc0 addr v1:172.21.15.18:6789/0 2024-03-29T11:09:51.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.276+0000 7f6b8f1c7640 10 monclient(hunting): picked mon.noname-b con 0x7f6b8814ff40 addr v1:172.21.15.70:6789/0 2024-03-29T11:09:51.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.276+0000 7f6b8f1c7640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b8814aae0 con 0x7f6b88149dc0 2024-03-29T11:09:51.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.276+0000 7f6b8f1c7640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b88070520 con 0x7f6b8814ff40 2024-03-29T11:09:51.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.276+0000 7f6b8f1c7640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b88070800 con 0x7f6b880a9930 2024-03-29T11:09:51.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.276+0000 7f6b8f1c7640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:51.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.276+0000 7f6b8f1c7640 10 monclient(hunting): authenticate will time out at 817.548026s 2024-03-29T11:09:51.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.279+0000 7f6b8d764640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f6b8814ff40 0x7f6b8812f400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.18:54475/0 (socket says 172.21.15.18:54475) 2024-03-29T11:09:51.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.279+0000 7f6b8d764640 1 -- 172.21.15.18:0/3240185147 learned_addr learned my addr 172.21.15.18:0/3240185147 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:09:51.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.280+0000 7f6b857fa640 10 client.?.objecter ms_handle_connect 0x7f6b8814ff40 2024-03-29T11:09:51.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.280+0000 7f6b857fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:51.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.282+0000 7f6b857fa640 1 -- 172.21.15.18:0/3240185147 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 17922548 0 0) 0x7f6b88070520 con 0x7f6b8814ff40 2024-03-29T11:09:51.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.282+0000 7f6b857fa640 10 monclient(hunting): _negotiate 2024-03-29T11:09:51.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.282+0000 7f6b857fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:51.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.282+0000 7f6b857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:51.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.282+0000 7f6b857fa640 10 monclient(hunting): my global_id is 4258 2024-03-29T11:09:51.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.282+0000 7f6b857fa640 1 -- 172.21.15.18:0/3240185147 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b74004eb0 con 0x7f6b8814ff40 2024-03-29T11:09:51.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.282+0000 7f6b857fa640 10 client.?.objecter ms_handle_connect 0x7f6b88149dc0 2024-03-29T11:09:51.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.282+0000 7f6b857fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:51.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.283+0000 7f6b857fa640 1 -- 172.21.15.18:0/3240185147 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3468119070 0 0) 0x7f6b8814aae0 con 0x7f6b88149dc0 2024-03-29T11:09:51.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.283+0000 7f6b857fa640 10 monclient(hunting): _negotiate 2024-03-29T11:09:51.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.283+0000 7f6b857fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:51.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.283+0000 7f6b857fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:51.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.283+0000 7f6b857fa640 10 monclient(hunting): my global_id is 4263 2024-03-29T11:09:51.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.283+0000 7f6b857fa640 1 -- 172.21.15.18:0/3240185147 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b88070520 con 0x7f6b88149dc0 2024-03-29T11:09:51.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.283+0000 7f6b857fa640 1 -- 172.21.15.18:0/3240185147 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1970295932 0 0) 0x7f6b88070520 con 0x7f6b88149dc0 2024-03-29T11:09:51.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.283+0000 7f6b857fa640 1 -- 172.21.15.18:0/3240185147 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6b8814aae0 con 0x7f6b88149dc0 2024-03-29T11:09:51.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.283+0000 7f6b857fa640 1 -- 172.21.15.18:0/3240185147 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f6b7c001ae0 con 0x7f6b88149dc0 2024-03-29T11:09:51.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.283+0000 7f6b857fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:51.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.283+0000 7f6b857fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:09:51.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.283+0000 7f6b857fa640 10 monclient(hunting): dump: 2024-03-29T11:09:51.264 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:09:51.264 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:51.264 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:51.264 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:51.264 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:51.264 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:51.264 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:51.264 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:51.265 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:51.265 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:51.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.283+0000 7f6b857fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:09:51.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.283+0000 7f6b857fa640 1 -- 172.21.15.18:0/3240185147 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1469085858 0 0) 0x7f6b8814aae0 con 0x7f6b88149dc0 2024-03-29T11:09:51.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.283+0000 7f6b857fa640 1 -- 172.21.15.18:0/3240185147 >> v1:172.21.15.18:6790/0 conn(0x7f6b880a9930 legacy=0x7f6b8812daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:51.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.283+0000 7f6b857fa640 1 -- 172.21.15.18:0/3240185147 >> v1:172.21.15.70:6789/0 conn(0x7f6b8814ff40 legacy=0x7f6b8812f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:51.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.283+0000 7f6b857fa640 10 monclient: _finish_hunting 0 2024-03-29T11:09:51.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.283+0000 7f6b857fa640 1 monclient: found mon.a 2024-03-29T11:09:51.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.283+0000 7f6b857fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:51.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.283+0000 7f6b857fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:09:51.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.283+0000 7f6b857fa640 1 -- 172.21.15.18:0/3240185147 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b88070ae0 con 0x7f6b88149dc0 2024-03-29T11:09:51.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.283+0000 7f6b857fa640 10 monclient: _finish_auth 0 2024-03-29T11:09:51.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.283+0000 7f6b857fa640 10 monclient: _check_auth_tickets 2024-03-29T11:09:51.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.283+0000 7f6b857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:51.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.283+0000 7f6b8f1c7640 5 monclient: authenticate success, global_id 4263 2024-03-29T11:09:51.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.283+0000 7f6b8f1c7640 10 monclient: _renew_subs 2024-03-29T11:09:51.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.283+0000 7f6b8f1c7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:09:51.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.283+0000 7f6b8f1c7640 1 -- 172.21.15.18:0/3240185147 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b8806f360 con 0x7f6b88149dc0 2024-03-29T11:09:51.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.283+0000 7f6b8f1c7640 10 client.4263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:09:51.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.283+0000 7f6b8f1c7640 10 monclient: _renew_subs 2024-03-29T11:09:51.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.283+0000 7f6b8f1c7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:09:51.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.283+0000 7f6b8f1c7640 1 -- 172.21.15.18:0/3240185147 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b8806f880 con 0x7f6b88149dc0 2024-03-29T11:09:51.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.283+0000 7f6b8f1c7640 1 librados: init done 2024-03-29T11:09:51.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.284+0000 7f6b857fa640 1 -- 172.21.15.18:0/3240185147 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6b7c0038a0 con 0x7f6b88149dc0 2024-03-29T11:09:51.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.284+0000 7f6b857fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:51.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.284+0000 7f6b857fa640 1 -- 172.21.15.18:0/3240185147 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f6b7c003bb0 con 0x7f6b88149dc0 2024-03-29T11:09:51.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.284+0000 7f6b857fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:51.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.284+0000 7f6b857fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:09:51.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.284+0000 7f6b857fa640 10 monclient: dump: 2024-03-29T11:09:51.267 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:09:51.267 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:51.267 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:51.267 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:51.267 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:51.268 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:51.268 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:51.268 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:51.268 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:51.268 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:51.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.284+0000 7f6b86ffd640 4 set_mon_vals no callback set 2024-03-29T11:09:51.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.286+0000 7f6b857fa640 1 -- 172.21.15.18:0/3240185147 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f6b7c019b80 con 0x7f6b88149dc0 2024-03-29T11:09:51.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.287+0000 7f6b857fa640 1 -- 172.21.15.18:0/3240185147 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f6b7c083910 con 0x7f6b88149dc0 2024-03-29T11:09:51.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.287+0000 7f6b8f1c7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:51.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.287+0000 7f6b857fa640 10 client.4263.objecter ms_dispatch 0x7f6b88000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:09:51.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.287+0000 7f6b857fa640 3 client.4263.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:09:51.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.287+0000 7f6b857fa640 3 client.4263.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:09:51.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.287+0000 7f6b8f1c7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:51.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.287+0000 7f6b8f1c7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:09:51.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.287+0000 7f6b857fa640 20 client.4263.objecter dump_active .. 0 homeless 2024-03-29T11:09:51.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.287+0000 7f6b8f1c7640 1 -- 172.21.15.18:0/3240185147 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b68005080 con 0x7f6b88149dc0 2024-03-29T11:09:51.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.291+0000 7f6b8cf63640 10 monclient: get_auth_request con 0x7f6b7406dd70 auth_method 0 2024-03-29T11:09:51.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.291+0000 7f6b857fa640 10 client.4263.objecter ms_handle_connect 0x7f6b7406dd70 2024-03-29T11:09:51.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.297+0000 7f6b857fa640 1 -- 172.21.15.18:0/3240185147 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6b7c04f1f0 con 0x7f6b88149dc0 2024-03-29T11:09:51.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.297+0000 7f6b857fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:51.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.297+0000 7f6b857fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:09:51.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.609+0000 7f6b8f1c7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:51.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.609+0000 7f6b8f1c7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:51.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.609+0000 7f6b8f1c7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:09:51.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.609+0000 7f6b8f1c7640 1 -- 172.21.15.18:0/3240185147 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6b68005330 con 0x7f6b88149dc0 2024-03-29T11:09:51.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.613+0000 7f6b857fa640 1 -- 172.21.15.18:0/3240185147 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f6b7c051120 con 0x7f6b88149dc0 2024-03-29T11:09:51.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.613+0000 7f6b857fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:51.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.613+0000 7f6b857fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:09:51.594 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:09:51.594 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:09:51.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.616+0000 7f6b8f1c7640 10 librados: watch_flush enter 2024-03-29T11:09:51.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.617+0000 7f6b8f1c7640 10 librados: watch_flush exit 2024-03-29T11:09:51.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.617+0000 7f6b8f1c7640 20 client.4263.objecter shutdown clearing up homeless session... 2024-03-29T11:09:51.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.617+0000 7f6b8f1c7640 10 client.4263.objecter successfully canceled tick 2024-03-29T11:09:51.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.617+0000 7f6b8f1c7640 1 -- 172.21.15.18:0/3240185147 >> v1:172.21.15.70:6816/3101871562 conn(0x7f6b7406dd70 legacy=0x7f6b74070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:51.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.617+0000 7f6b8f1c7640 10 monclient: shutdown 2024-03-29T11:09:51.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.617+0000 7f6b8f1c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:51.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.617+0000 7f6b8f1c7640 1 -- 172.21.15.18:0/3240185147 >> v1:172.21.15.18:6789/0 conn(0x7f6b88149dc0 legacy=0x7f6b8812ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:51.596 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.617+0000 7f6b8f1c7640 1 -- 172.21.15.18:0/3240185147 shutdown_connections 2024-03-29T11:09:51.596 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.617+0000 7f6b8f1c7640 1 -- 172.21.15.18:0/3240185147 >> 172.21.15.18:0/3240185147 conn(0x7f6b880b35e0 msgr2=0x7f6b880b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:51.596 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.618+0000 7f6b8f1c7640 1 -- 172.21.15.18:0/3240185147 shutdown_connections 2024-03-29T11:09:51.596 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.618+0000 7f6b8f1c7640 1 -- 172.21.15.18:0/3240185147 wait complete. 2024-03-29T11:09:51.597 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:51.618+0000 7f6b8f1c7640 1 librados: shutdown 2024-03-29T11:09:51.619 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:09:58.425 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:09:58.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.597+0000 7f26a20c3640 10 monclient: get_monmap_and_config 2024-03-29T11:09:58.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.597+0000 7f26a20c3640 10 monclient: build_initial_monmap 2024-03-29T11:09:58.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.597+0000 7f26a20c3640 10 monclient: monmap: 2024-03-29T11:09:58.576 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:09:58.576 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:58.576 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:09:58.597979+0000 2024-03-29T11:09:58.576 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:09:58.597979+0000 2024-03-29T11:09:58.576 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:58.576 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:58.576 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:09:58.576 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:09:58.577 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:09:58.577 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:58.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.598+0000 7f26a20c3640 1 Processor -- start 2024-03-29T11:09:58.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.599+0000 7f26a20c3640 1 -- start start 2024-03-29T11:09:58.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.599+0000 7f26a20c3640 10 monclient: init 2024-03-29T11:09:58.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.600+0000 7f26a20c3640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:58.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.600+0000 7f26a20c3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:09:58.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.600+0000 7f26a20c3640 10 monclient(hunting): picked mon.noname-b con 0x7f2694006410 addr v1:172.21.15.18:6790/0 2024-03-29T11:09:58.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.600+0000 7f26a20c3640 10 monclient(hunting): picked mon.noname-c con 0x7f269402a640 addr v1:172.21.15.70:6789/0 2024-03-29T11:09:58.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.600+0000 7f26a20c3640 10 monclient(hunting): picked mon.noname-a con 0x7f26940327d0 addr v1:172.21.15.18:6789/0 2024-03-29T11:09:58.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.600+0000 7f26a20c3640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f269402c820 con 0x7f26940327d0 2024-03-29T11:09:58.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.600+0000 7f26a20c3640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f269402a120 con 0x7f269402a640 2024-03-29T11:09:58.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.600+0000 7f26a20c3640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26940c75b0 con 0x7f2694006410 2024-03-29T11:09:58.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.600+0000 7f26a20c3640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:58.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.600+0000 7f26a20c3640 10 monclient(hunting): authenticate will time out at 824.872216s 2024-03-29T11:09:58.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.603+0000 7f26a08c0640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f269402a640 0x7f26940320c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.18:41863/0 (socket says 172.21.15.18:41863) 2024-03-29T11:09:58.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.603+0000 7f26a08c0640 1 -- 172.21.15.18:0/3560390981 learned_addr learned my addr 172.21.15.18:0/3560390981 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:09:58.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.605+0000 7f2693fff640 1 -- 172.21.15.18:0/3560390981 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3150505304 0 0) 0x7f269402a120 con 0x7f269402a640 2024-03-29T11:09:58.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.605+0000 7f2693fff640 10 monclient(hunting): _negotiate 2024-03-29T11:09:58.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.605+0000 7f2693fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:58.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.605+0000 7f2693fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:58.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.605+0000 7f2693fff640 10 monclient(hunting): my global_id is 4261 2024-03-29T11:09:58.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.605+0000 7f2693fff640 1 -- 172.21.15.18:0/3560390981 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2688003520 con 0x7f269402a640 2024-03-29T11:09:58.585 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.606+0000 7f2693fff640 1 -- 172.21.15.18:0/3560390981 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 69809410 0 0) 0x7f2688003520 con 0x7f269402a640 2024-03-29T11:09:58.585 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.606+0000 7f2693fff640 1 -- 172.21.15.18:0/3560390981 >> v1:172.21.15.18:6790/0 conn(0x7f2694006410 legacy=0x7f26940c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:09:58.585 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.606+0000 7f2693fff640 1 -- 172.21.15.18:0/3560390981 >> v1:172.21.15.18:6789/0 conn(0x7f26940327d0 legacy=0x7f26940c8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:09:58.585 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.606+0000 7f2693fff640 10 monclient: _finish_hunting 0 2024-03-29T11:09:58.585 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.606+0000 7f2693fff640 1 monclient: found mon.noname-c 2024-03-29T11:09:58.585 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.606+0000 7f2693fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:58.585 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.606+0000 7f2693fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.70:6789/0 2024-03-29T11:09:58.585 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.606+0000 7f2693fff640 1 -- 172.21.15.18:0/3560390981 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2694033110 con 0x7f269402a640 2024-03-29T11:09:58.586 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.606+0000 7f2693fff640 10 monclient: _finish_auth 0 2024-03-29T11:09:58.586 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.606+0000 7f2693fff640 10 monclient: _check_auth_tickets 2024-03-29T11:09:58.586 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.606+0000 7f2693fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:58.586 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.606+0000 7f2693fff640 1 -- 172.21.15.18:0/3560390981 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f269c04f730 con 0x7f269402a640 2024-03-29T11:09:58.586 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.606+0000 7f2693fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:58.586 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.606+0000 7f2693fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:09:58.586 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.606+0000 7f2693fff640 10 monclient: dump: 2024-03-29T11:09:58.586 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:09:58.586 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:58.586 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:58.586 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:58.586 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:58.586 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:58.586 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:58.587 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:58.587 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:58.587 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:58.587 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.606+0000 7f26a20c3640 5 monclient: authenticate success, global_id 4261 2024-03-29T11:09:58.587 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.606+0000 7f26a20c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:09:58.587 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.607+0000 7f2693fff640 1 -- 172.21.15.18:0/3560390981 <== mon.1 v1:172.21.15.70:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f269c075f10 con 0x7f269402a640 2024-03-29T11:09:58.587 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.607+0000 7f2693fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:58.587 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.607+0000 7f2693fff640 1 -- 172.21.15.18:0/3560390981 <== mon.1 v1:172.21.15.70:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f269c0758b0 con 0x7f269402a640 2024-03-29T11:09:58.587 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.607+0000 7f2693fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:58.587 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.607+0000 7f2693fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:09:58.587 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.607+0000 7f2693fff640 10 monclient: dump: 2024-03-29T11:09:58.587 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:09:58.587 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:58.587 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:58.588 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:58.588 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:58.588 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:58.588 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:58.588 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:58.588 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:58.588 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:58.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.607+0000 7f26a20c3640 10 monclient: get_monmap_and_config success 2024-03-29T11:09:58.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.607+0000 7f26a20c3640 4 set_mon_vals no callback set 2024-03-29T11:09:58.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.609+0000 7f26a20c3640 10 monclient: shutdown 2024-03-29T11:09:58.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.609+0000 7f26a20c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:58.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.609+0000 7f26a20c3640 1 -- 172.21.15.18:0/3560390981 >> v1:172.21.15.70:6789/0 conn(0x7f269402a640 legacy=0x7f26940320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:58.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.612+0000 7f26a20c3640 1 -- 172.21.15.18:0/3560390981 shutdown_connections 2024-03-29T11:09:58.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.612+0000 7f26a20c3640 1 -- 172.21.15.18:0/3560390981 >> 172.21.15.18:0/3560390981 conn(0x7f26940058f0 msgr2=0x7f2694005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:58.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.612+0000 7f26a20c3640 1 -- 172.21.15.18:0/3560390981 shutdown_connections 2024-03-29T11:09:58.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.612+0000 7f26a20c3640 1 -- 172.21.15.18:0/3560390981 wait complete. 2024-03-29T11:09:58.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.613+0000 7f26a20c3640 10 monclient: build_initial_monmap 2024-03-29T11:09:58.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.613+0000 7f26a20c3640 10 monclient: monmap: 2024-03-29T11:09:58.592 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:09:58.592 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:09:58.592 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:09:58.592 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:09:58.592 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:09:58.592 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:58.592 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:09:58.592 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:09:58.592 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:09:58.592 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:58.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.613+0000 7f26a20c3640 1 librados: starting msgr at 2024-03-29T11:09:58.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.613+0000 7f26a20c3640 1 librados: starting objecter 2024-03-29T11:09:58.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.614+0000 7f26a20c3640 1 Processor -- start 2024-03-29T11:09:58.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.614+0000 7f26a20c3640 1 -- start start 2024-03-29T11:09:58.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.614+0000 7f26a20c3640 1 librados: setting wanted keys 2024-03-29T11:09:58.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.614+0000 7f26a20c3640 1 librados: calling monclient init 2024-03-29T11:09:58.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.614+0000 7f26a20c3640 10 monclient: init 2024-03-29T11:09:58.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.614+0000 7f26a20c3640 10 monclient: _reopen_session rank -1 2024-03-29T11:09:58.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.614+0000 7f26a20c3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:09:58.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.617+0000 7f26a20c3640 10 monclient(hunting): picked mon.noname-a con 0x7f2694006410 addr v1:172.21.15.18:6789/0 2024-03-29T11:09:58.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.617+0000 7f26a20c3640 10 monclient(hunting): picked mon.noname-b con 0x7f269402a640 addr v1:172.21.15.70:6789/0 2024-03-29T11:09:58.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.617+0000 7f26a20c3640 10 monclient(hunting): picked mon.noname-c con 0x7f26940327d0 addr v1:172.21.15.18:6790/0 2024-03-29T11:09:58.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.617+0000 7f26a20c3640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26940314f0 con 0x7f2694006410 2024-03-29T11:09:58.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.617+0000 7f26a20c3640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2694030f80 con 0x7f269402a640 2024-03-29T11:09:58.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.617+0000 7f26a20c3640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26940bb3e0 con 0x7f26940327d0 2024-03-29T11:09:58.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.617+0000 7f26a20c3640 10 monclient(hunting): _renew_subs 2024-03-29T11:09:58.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.617+0000 7f26a20c3640 10 monclient(hunting): authenticate will time out at 824.889784s 2024-03-29T11:09:58.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.620+0000 7f26a10c1640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f2694006410 0x7f26940c0260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:55357/0 (socket says 172.21.15.18:55357) 2024-03-29T11:09:58.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.620+0000 7f26a10c1640 1 -- 172.21.15.18:0/2157610318 learned_addr learned my addr 172.21.15.18:0/2157610318 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:09:58.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.620+0000 7f2691ffb640 10 client.?.objecter ms_handle_connect 0x7f2694006410 2024-03-29T11:09:58.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.620+0000 7f2691ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:58.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.620+0000 7f2691ffb640 1 -- 172.21.15.18:0/2157610318 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4270185867 0 0) 0x7f26940314f0 con 0x7f2694006410 2024-03-29T11:09:58.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.621+0000 7f2691ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:09:58.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.621+0000 7f2691ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:09:58.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.621+0000 7f2691ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:09:58.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.621+0000 7f2691ffb640 10 monclient(hunting): my global_id is 4266 2024-03-29T11:09:58.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.621+0000 7f2691ffb640 1 -- 172.21.15.18:0/2157610318 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2680004eb0 con 0x7f2694006410 2024-03-29T11:09:58.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.621+0000 7f2691ffb640 10 client.?.objecter ms_handle_connect 0x7f269402a640 2024-03-29T11:09:58.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.621+0000 7f2691ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:09:58.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.621+0000 7f2691ffb640 1 -- 172.21.15.18:0/2157610318 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1051785241 0 0) 0x7f2680004eb0 con 0x7f2694006410 2024-03-29T11:09:58.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.621+0000 7f2691ffb640 1 -- 172.21.15.18:0/2157610318 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26940314f0 con 0x7f2694006410 2024-03-29T11:09:58.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.622+0000 7f2691ffb640 1 -- 172.21.15.18:0/2157610318 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f268c004080 con 0x7f2694006410 2024-03-29T11:09:58.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.622+0000 7f2691ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:58.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.622+0000 7f2691ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:09:58.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.622+0000 7f2691ffb640 10 monclient(hunting): dump: 2024-03-29T11:09:58.604 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:09:58.604 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:58.604 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:58.604 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:58.604 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:58.604 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:58.605 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:58.605 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:58.605 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:58.605 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:58.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.622+0000 7f2691ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:09:58.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.622+0000 7f2691ffb640 1 -- 172.21.15.18:0/2157610318 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4167831910 0 0) 0x7f26940314f0 con 0x7f2694006410 2024-03-29T11:09:58.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.622+0000 7f2691ffb640 1 -- 172.21.15.18:0/2157610318 >> v1:172.21.15.18:6790/0 conn(0x7f26940327d0 legacy=0x7f2694167710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:09:58.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.624+0000 7f2691ffb640 1 -- 172.21.15.18:0/2157610318 >> v1:172.21.15.70:6789/0 conn(0x7f269402a640 legacy=0x7f26940bacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:58.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.624+0000 7f2691ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:09:58.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.624+0000 7f2691ffb640 1 monclient: found mon.a 2024-03-29T11:09:58.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.624+0000 7f2691ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:09:58.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.624+0000 7f2691ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:09:58.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.624+0000 7f2691ffb640 1 -- 172.21.15.18:0/2157610318 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f269402e2f0 con 0x7f2694006410 2024-03-29T11:09:58.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.624+0000 7f2691ffb640 10 monclient: _finish_auth 0 2024-03-29T11:09:58.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.624+0000 7f2691ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:09:58.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.624+0000 7f2691ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:09:58.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.624+0000 7f26a20c3640 5 monclient: authenticate success, global_id 4266 2024-03-29T11:09:58.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.624+0000 7f26a20c3640 10 monclient: _renew_subs 2024-03-29T11:09:58.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.624+0000 7f26a20c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:09:58.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.624+0000 7f26a20c3640 1 -- 172.21.15.18:0/2157610318 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2694031980 con 0x7f2694006410 2024-03-29T11:09:58.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.625+0000 7f26a20c3640 10 client.4266.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:09:58.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.625+0000 7f26a20c3640 10 monclient: _renew_subs 2024-03-29T11:09:58.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.625+0000 7f26a20c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:09:58.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.625+0000 7f26a20c3640 1 -- 172.21.15.18:0/2157610318 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f26940bd770 con 0x7f2694006410 2024-03-29T11:09:58.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.625+0000 7f26a20c3640 1 librados: init done 2024-03-29T11:09:58.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.625+0000 7f2691ffb640 1 -- 172.21.15.18:0/2157610318 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f268c0044d0 con 0x7f2694006410 2024-03-29T11:09:58.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.625+0000 7f2691ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:09:58.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.625+0000 7f2691ffb640 1 -- 172.21.15.18:0/2157610318 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f268c001180 con 0x7f2694006410 2024-03-29T11:09:58.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.625+0000 7f2691ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:09:58.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.625+0000 7f2691ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:09:58.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.625+0000 7f26937fe640 4 set_mon_vals no callback set 2024-03-29T11:09:58.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.625+0000 7f2691ffb640 10 monclient: dump: 2024-03-29T11:09:58.609 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:09:58.609 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:09:58.609 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:58.609 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:09:58.609 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:09:58.609 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:09:58.609 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:09:58.609 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:09:58.609 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:09:58.609 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:09:58.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.626+0000 7f26a20c3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:58.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.626+0000 7f26a20c3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:58.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.626+0000 7f26a20c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:09:58.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.626+0000 7f26a20c3640 1 -- 172.21.15.18:0/2157610318 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f267c005080 con 0x7f2694006410 2024-03-29T11:09:58.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.627+0000 7f2691ffb640 1 -- 172.21.15.18:0/2157610318 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f268c0013a0 con 0x7f2694006410 2024-03-29T11:09:58.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.628+0000 7f2691ffb640 1 -- 172.21.15.18:0/2157610318 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f268c084a50 con 0x7f2694006410 2024-03-29T11:09:58.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.628+0000 7f2691ffb640 10 client.4266.objecter ms_dispatch 0x7f269c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:09:58.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.628+0000 7f2691ffb640 3 client.4266.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:09:58.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.628+0000 7f2691ffb640 3 client.4266.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:09:58.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.628+0000 7f2691ffb640 20 client.4266.objecter dump_active .. 0 homeless 2024-03-29T11:09:58.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.628+0000 7f26a08c0640 10 monclient: get_auth_request con 0x7f268006dbc0 auth_method 0 2024-03-29T11:09:58.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.629+0000 7f2691ffb640 10 client.4266.objecter ms_handle_connect 0x7f268006dbc0 2024-03-29T11:09:58.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.637+0000 7f2691ffb640 1 -- 172.21.15.18:0/2157610318 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f268c050160 con 0x7f2694006410 2024-03-29T11:09:58.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.637+0000 7f2691ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:09:58.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.637+0000 7f2691ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:09:58.952 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:09:58.952 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:09:58.952 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.971+0000 7f26a20c3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:58.952 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.971+0000 7f26a20c3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:58.952 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.971+0000 7f26a20c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:09:58.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.971+0000 7f26a20c3640 1 -- 172.21.15.18:0/2157610318 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f267c005330 con 0x7f2694006410 2024-03-29T11:09:58.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.972+0000 7f2691ffb640 1 -- 172.21.15.18:0/2157610318 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f268c052260 con 0x7f2694006410 2024-03-29T11:09:58.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.972+0000 7f2691ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:09:58.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.972+0000 7f2691ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:09:58.954 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.975+0000 7f26a20c3640 10 librados: watch_flush enter 2024-03-29T11:09:58.954 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.976+0000 7f26a20c3640 10 librados: watch_flush exit 2024-03-29T11:09:58.954 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.976+0000 7f26a20c3640 20 client.4266.objecter shutdown clearing up homeless session... 2024-03-29T11:09:58.954 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.976+0000 7f26a20c3640 10 client.4266.objecter successfully canceled tick 2024-03-29T11:09:58.954 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.976+0000 7f26a20c3640 1 -- 172.21.15.18:0/2157610318 >> v1:172.21.15.70:6816/3101871562 conn(0x7f268006dbc0 legacy=0x7f2680070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:58.954 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.976+0000 7f26a20c3640 10 monclient: shutdown 2024-03-29T11:09:58.954 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.976+0000 7f26a20c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:09:58.954 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.976+0000 7f26a20c3640 1 -- 172.21.15.18:0/2157610318 >> v1:172.21.15.18:6789/0 conn(0x7f2694006410 legacy=0x7f26940c0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:09:58.955 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.976+0000 7f26a20c3640 1 -- 172.21.15.18:0/2157610318 shutdown_connections 2024-03-29T11:09:58.955 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.976+0000 7f26a20c3640 1 -- 172.21.15.18:0/2157610318 >> 172.21.15.18:0/2157610318 conn(0x7f26940058f0 msgr2=0x7f26940043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:09:58.955 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.977+0000 7f26a20c3640 1 -- 172.21.15.18:0/2157610318 shutdown_connections 2024-03-29T11:09:58.955 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.977+0000 7f26a20c3640 1 -- 172.21.15.18:0/2157610318 wait complete. 2024-03-29T11:09:58.955 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:09:58.977+0000 7f26a20c3640 1 librados: shutdown 2024-03-29T11:09:58.974 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:10:05.781 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:10:05.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.952+0000 7efec5e63640 10 monclient: get_monmap_and_config 2024-03-29T11:10:05.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.952+0000 7efec5e63640 10 monclient: build_initial_monmap 2024-03-29T11:10:05.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.952+0000 7efec5e63640 10 monclient: monmap: 2024-03-29T11:10:05.931 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:10:05.931 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:05.931 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:10:05.953862+0000 2024-03-29T11:10:05.931 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:10:05.953862+0000 2024-03-29T11:10:05.931 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:05.932 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:05.932 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:10:05.932 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:10:05.932 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:10:05.932 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:05.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.955+0000 7efec5e63640 1 Processor -- start 2024-03-29T11:10:05.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.955+0000 7efec5e63640 1 -- start start 2024-03-29T11:10:05.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.955+0000 7efec5e63640 10 monclient: init 2024-03-29T11:10:05.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.956+0000 7efec5e63640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:05.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.956+0000 7efec5e63640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:10:05.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.956+0000 7efec5e63640 10 monclient(hunting): picked mon.noname-c con 0x7efec00b1f60 addr v1:172.21.15.70:6789/0 2024-03-29T11:10:05.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.956+0000 7efec5e63640 10 monclient(hunting): picked mon.noname-b con 0x7efec0071eb0 addr v1:172.21.15.18:6790/0 2024-03-29T11:10:05.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.956+0000 7efec5e63640 10 monclient(hunting): picked mon.noname-a con 0x7efec0143d10 addr v1:172.21.15.18:6789/0 2024-03-29T11:10:05.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.956+0000 7efec5e63640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efec006ef80 con 0x7efec0143d10 2024-03-29T11:10:05.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.956+0000 7efec5e63640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efec006f260 con 0x7efec00b1f60 2024-03-29T11:10:05.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.956+0000 7efec5e63640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efec00b40c0 con 0x7efec0071eb0 2024-03-29T11:10:05.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.956+0000 7efec5e63640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:05.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.956+0000 7efec5e63640 10 monclient(hunting): authenticate will time out at 832.228077s 2024-03-29T11:10:05.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.960+0000 7efebeffd640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7efec0071eb0 0x7efec0146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:44167/0 (socket says 172.21.15.18:44167) 2024-03-29T11:10:05.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.960+0000 7efebeffd640 1 -- 172.21.15.18:0/35719213 learned_addr learned my addr 172.21.15.18:0/35719213 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:10:05.942 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.961+0000 7efebe7fc640 1 -- 172.21.15.18:0/35719213 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1824482180 0 0) 0x7efec00b40c0 con 0x7efec0071eb0 2024-03-29T11:10:05.942 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.961+0000 7efebe7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:10:05.942 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.961+0000 7efebe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:05.942 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.961+0000 7efebe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:05.942 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.961+0000 7efebe7fc640 10 monclient(hunting): my global_id is 4268 2024-03-29T11:10:05.942 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.961+0000 7efebe7fc640 1 -- 172.21.15.18:0/35719213 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efea8003520 con 0x7efec0071eb0 2024-03-29T11:10:05.942 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.961+0000 7efebe7fc640 1 -- 172.21.15.18:0/35719213 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1893122168 0 0) 0x7efea8003520 con 0x7efec0071eb0 2024-03-29T11:10:05.942 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.961+0000 7efebe7fc640 1 -- 172.21.15.18:0/35719213 >> v1:172.21.15.70:6789/0 conn(0x7efec00b1f60 legacy=0x7efec0143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:05.942 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.961+0000 7efebe7fc640 1 -- 172.21.15.18:0/35719213 >> v1:172.21.15.18:6789/0 conn(0x7efec0143d10 legacy=0x7efec0146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:05.942 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.961+0000 7efebe7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:10:05.942 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.961+0000 7efebe7fc640 1 monclient: found mon.noname-b 2024-03-29T11:10:05.942 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.961+0000 7efebe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:05.942 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.961+0000 7efebe7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:10:05.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.961+0000 7efebe7fc640 1 -- 172.21.15.18:0/35719213 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec0155d70 con 0x7efec0071eb0 2024-03-29T11:10:05.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.961+0000 7efebe7fc640 10 monclient: _finish_auth 0 2024-03-29T11:10:05.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.961+0000 7efebe7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:10:05.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.961+0000 7efebe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:05.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.962+0000 7efec5e63640 5 monclient: authenticate success, global_id 4268 2024-03-29T11:10:05.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.962+0000 7efec5e63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:10:05.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.962+0000 7efebe7fc640 1 -- 172.21.15.18:0/35719213 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7efeb4001690 con 0x7efec0071eb0 2024-03-29T11:10:05.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.962+0000 7efebe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:05.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.962+0000 7efebe7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:10:05.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.962+0000 7efebe7fc640 10 monclient: dump: 2024-03-29T11:10:05.943 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:10:05.943 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:05.943 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:05.944 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:05.944 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:05.944 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:05.944 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:10:05.944 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:10:05.944 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:10:05.944 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:05.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.962+0000 7efec5e63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:10:05.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.962+0000 7efebe7fc640 1 -- 172.21.15.18:0/35719213 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efeb40018b0 con 0x7efec0071eb0 2024-03-29T11:10:05.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.962+0000 7efebe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:05.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.962+0000 7efec5e63640 10 monclient: get_monmap_and_config success 2024-03-29T11:10:05.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.962+0000 7efec5e63640 4 set_mon_vals no callback set 2024-03-29T11:10:05.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.963+0000 7efec5e63640 10 monclient: shutdown 2024-03-29T11:10:05.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.963+0000 7efec5e63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:05.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.963+0000 7efec5e63640 1 -- 172.21.15.18:0/35719213 >> v1:172.21.15.18:6790/0 conn(0x7efec0071eb0 legacy=0x7efec0146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:05.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.965+0000 7efec5e63640 1 -- 172.21.15.18:0/35719213 shutdown_connections 2024-03-29T11:10:05.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.965+0000 7efec5e63640 1 -- 172.21.15.18:0/35719213 >> 172.21.15.18:0/35719213 conn(0x7efec0066a00 msgr2=0x7efec0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:05.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.966+0000 7efec5e63640 1 -- 172.21.15.18:0/35719213 shutdown_connections 2024-03-29T11:10:05.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.966+0000 7efec5e63640 1 -- 172.21.15.18:0/35719213 wait complete. 2024-03-29T11:10:05.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.968+0000 7efec5e63640 10 monclient: build_initial_monmap 2024-03-29T11:10:05.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.968+0000 7efec5e63640 10 monclient: monmap: 2024-03-29T11:10:05.950 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:10:05.950 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:10:05.950 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:10:05.950 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:10:05.950 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:05.950 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:05.950 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:10:05.950 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:10:05.951 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:10:05.951 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:05.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.968+0000 7efec5e63640 1 librados: starting msgr at 2024-03-29T11:10:05.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.968+0000 7efec5e63640 1 librados: starting objecter 2024-03-29T11:10:05.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.968+0000 7efec5e63640 1 Processor -- start 2024-03-29T11:10:05.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.968+0000 7efec5e63640 1 -- start start 2024-03-29T11:10:05.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.968+0000 7efec5e63640 1 librados: setting wanted keys 2024-03-29T11:10:05.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.968+0000 7efec5e63640 1 librados: calling monclient init 2024-03-29T11:10:05.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.968+0000 7efec5e63640 10 monclient: init 2024-03-29T11:10:05.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.969+0000 7efec5e63640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:05.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.969+0000 7efec5e63640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:10:05.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.969+0000 7efec5e63640 10 monclient(hunting): picked mon.noname-b con 0x7efec00b1f60 addr v1:172.21.15.70:6789/0 2024-03-29T11:10:05.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.969+0000 7efec5e63640 10 monclient(hunting): picked mon.noname-c con 0x7efec0143d10 addr v1:172.21.15.18:6790/0 2024-03-29T11:10:05.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.969+0000 7efec5e63640 10 monclient(hunting): picked mon.noname-a con 0x7efec006fbd0 addr v1:172.21.15.18:6789/0 2024-03-29T11:10:05.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.969+0000 7efec5e63640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efec0152b60 con 0x7efec006fbd0 2024-03-29T11:10:05.952 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.969+0000 7efec5e63640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efec00ac810 con 0x7efec00b1f60 2024-03-29T11:10:05.952 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.969+0000 7efec5e63640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efec00acaf0 con 0x7efec0143d10 2024-03-29T11:10:05.952 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.969+0000 7efec5e63640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:05.952 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.969+0000 7efec5e63640 10 monclient(hunting): authenticate will time out at 832.241654s 2024-03-29T11:10:05.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.973+0000 7efebffff640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7efec006fbd0 0x7efec01ffc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:53109/0 (socket says 172.21.15.18:53109) 2024-03-29T11:10:05.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.973+0000 7efebffff640 1 -- 172.21.15.18:0/4061674981 learned_addr learned my addr 172.21.15.18:0/4061674981 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:10:05.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.974+0000 7efea7fff640 10 client.?.objecter ms_handle_connect 0x7efec006fbd0 2024-03-29T11:10:05.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.974+0000 7efea7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:05.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.974+0000 7efea7fff640 1 -- 172.21.15.18:0/4061674981 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1018472273 0 0) 0x7efec0152b60 con 0x7efec006fbd0 2024-03-29T11:10:05.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.974+0000 7efea7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:10:05.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.974+0000 7efea7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:05.954 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.974+0000 7efea7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:05.954 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.974+0000 7efea7fff640 10 monclient(hunting): my global_id is 4272 2024-03-29T11:10:05.954 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.974+0000 7efea7fff640 1 -- 172.21.15.18:0/4061674981 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efeac004eb0 con 0x7efec006fbd0 2024-03-29T11:10:05.954 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.976+0000 7efea7fff640 10 client.?.objecter ms_handle_connect 0x7efec0143d10 2024-03-29T11:10:05.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.976+0000 7efea7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:05.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.976+0000 7efea7fff640 1 -- 172.21.15.18:0/4061674981 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 86196081 0 0) 0x7efec00acaf0 con 0x7efec0143d10 2024-03-29T11:10:05.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.976+0000 7efea7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:10:05.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.976+0000 7efea7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:05.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.976+0000 7efea7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:05.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.976+0000 7efea7fff640 10 monclient(hunting): my global_id is 4271 2024-03-29T11:10:05.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.976+0000 7efea7fff640 1 -- 172.21.15.18:0/4061674981 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efec0152b60 con 0x7efec0143d10 2024-03-29T11:10:05.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.978+0000 7efea7fff640 1 -- 172.21.15.18:0/4061674981 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1424695504 0 0) 0x7efec0152b60 con 0x7efec0143d10 2024-03-29T11:10:05.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.979+0000 7efea7fff640 1 -- 172.21.15.18:0/4061674981 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efec00acaf0 con 0x7efec0143d10 2024-03-29T11:10:05.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.979+0000 7efea7fff640 1 -- 172.21.15.18:0/4061674981 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2523933573 0 0) 0x7efeac004eb0 con 0x7efec006fbd0 2024-03-29T11:10:05.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.979+0000 7efea7fff640 1 -- 172.21.15.18:0/4061674981 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efec0152b60 con 0x7efec006fbd0 2024-03-29T11:10:05.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.979+0000 7efea7fff640 1 -- 172.21.15.18:0/4061674981 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7efeb4004940 con 0x7efec0143d10 2024-03-29T11:10:05.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.979+0000 7efea7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:05.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.979+0000 7efea7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:10:05.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.979+0000 7efea7fff640 10 monclient(hunting): dump: 2024-03-29T11:10:05.962 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:10:05.962 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:05.962 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:05.962 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:05.962 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:05.962 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:05.962 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:10:05.962 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:10:05.962 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:10:05.962 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:05.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.979+0000 7efea7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:10:05.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.979+0000 7efea7fff640 10 client.?.objecter ms_handle_connect 0x7efec00b1f60 2024-03-29T11:10:05.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.979+0000 7efea7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:05.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.979+0000 7efea7fff640 1 -- 172.21.15.18:0/4061674981 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7efeb0001870 con 0x7efec006fbd0 2024-03-29T11:10:05.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.979+0000 7efea7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:05.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.979+0000 7efea7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:10:05.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.979+0000 7efea7fff640 10 monclient(hunting): dump: 2024-03-29T11:10:05.963 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:10:05.963 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:05.963 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:05.963 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:05.963 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:05.963 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:05.963 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:10:05.963 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:10:05.963 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:10:05.964 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:05.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.979+0000 7efea7fff640 1 -- 172.21.15.18:0/4061674981 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3131435109 0 0) 0x7efec00acaf0 con 0x7efec0143d10 2024-03-29T11:10:05.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.979+0000 7efea7fff640 1 -- 172.21.15.18:0/4061674981 >> v1:172.21.15.70:6789/0 conn(0x7efec00b1f60 legacy=0x7efec00aaf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:05.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.979+0000 7efea7fff640 1 -- 172.21.15.18:0/4061674981 >> v1:172.21.15.18:6789/0 conn(0x7efec006fbd0 legacy=0x7efec01ffc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:05.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.979+0000 7efea7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:10:05.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.979+0000 7efea7fff640 1 monclient: found mon.c 2024-03-29T11:10:05.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.979+0000 7efea7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:05.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.979+0000 7efea7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:10:05.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.979+0000 7efea7fff640 1 -- 172.21.15.18:0/4061674981 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec0138ba0 con 0x7efec0143d10 2024-03-29T11:10:05.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.979+0000 7efea7fff640 10 monclient: _finish_auth 0 2024-03-29T11:10:05.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.979+0000 7efea7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:10:05.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.979+0000 7efea7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:05.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.979+0000 7efec5e63640 5 monclient: authenticate success, global_id 4271 2024-03-29T11:10:05.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.979+0000 7efec5e63640 10 monclient: _renew_subs 2024-03-29T11:10:05.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.979+0000 7efec5e63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:10:05.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.979+0000 7efec5e63640 1 -- 172.21.15.18:0/4061674981 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efec00aa420 con 0x7efec0143d10 2024-03-29T11:10:05.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.979+0000 7efec5e63640 10 client.4271.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:10:05.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.979+0000 7efec5e63640 10 monclient: _renew_subs 2024-03-29T11:10:05.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.979+0000 7efec5e63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:10:05.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.979+0000 7efec5e63640 1 -- 172.21.15.18:0/4061674981 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efec00ab840 con 0x7efec0143d10 2024-03-29T11:10:05.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.979+0000 7efec5e63640 1 librados: init done 2024-03-29T11:10:05.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.981+0000 7efea7fff640 1 -- 172.21.15.18:0/4061674981 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efeb4004e80 con 0x7efec0143d10 2024-03-29T11:10:05.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.981+0000 7efea7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:05.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.981+0000 7efebdffb640 4 set_mon_vals no callback set 2024-03-29T11:10:05.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.981+0000 7efea7fff640 1 -- 172.21.15.18:0/4061674981 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7efeb4001250 con 0x7efec0143d10 2024-03-29T11:10:05.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.981+0000 7efea7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:05.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.981+0000 7efea7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:10:05.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.981+0000 7efea7fff640 10 monclient: dump: 2024-03-29T11:10:05.965 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:10:05.965 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:05.966 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:05.966 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:05.966 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:05.966 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:05.966 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:10:05.966 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:10:05.966 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:10:05.966 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:05.966 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.981+0000 7efec5e63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:05.966 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.981+0000 7efea7fff640 1 -- 172.21.15.18:0/4061674981 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7efeb4004ff0 con 0x7efec0143d10 2024-03-29T11:10:05.966 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.982+0000 7efea7fff640 1 -- 172.21.15.18:0/4061674981 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7efeb4084b30 con 0x7efec0143d10 2024-03-29T11:10:05.966 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.982+0000 7efea7fff640 10 client.4271.objecter ms_dispatch 0x7efec0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:10:05.966 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.982+0000 7efea7fff640 3 client.4271.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:10:05.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.982+0000 7efea7fff640 3 client.4271.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:10:05.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.982+0000 7efea7fff640 20 client.4271.objecter dump_active .. 0 homeless 2024-03-29T11:10:05.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.985+0000 7efec5e63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:05.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.985+0000 7efec5e63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:10:05.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.985+0000 7efec5e63640 1 -- 172.21.15.18:0/4061674981 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efea0005080 con 0x7efec0143d10 2024-03-29T11:10:05.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.986+0000 7efebf7fe640 10 monclient: get_auth_request con 0x7efeac06e120 auth_method 0 2024-03-29T11:10:05.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.986+0000 7efea7fff640 10 client.4271.objecter ms_handle_connect 0x7efeac06e120 2024-03-29T11:10:05.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.996+0000 7efea7fff640 1 -- 172.21.15.18:0/4061674981 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efeb4050590 con 0x7efec0143d10 2024-03-29T11:10:05.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.996+0000 7efea7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:05.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:05.996+0000 7efea7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:10:06.301 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:06.323+0000 7efec5e63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:06.301 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:06.323+0000 7efec5e63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:06.301 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:06.323+0000 7efec5e63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:10:06.301 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:06.323+0000 7efec5e63640 1 -- 172.21.15.18:0/4061674981 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efea0005330 con 0x7efec0143d10 2024-03-29T11:10:06.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:06.325+0000 7efea7fff640 1 -- 172.21.15.18:0/4061674981 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7efeb4052440 con 0x7efec0143d10 2024-03-29T11:10:06.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:06.325+0000 7efea7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:06.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:06.325+0000 7efea7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:10:06.304 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:10:06.304 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:10:06.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:06.330+0000 7efec5e63640 10 librados: watch_flush enter 2024-03-29T11:10:06.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:06.330+0000 7efec5e63640 10 librados: watch_flush exit 2024-03-29T11:10:06.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:06.330+0000 7efec5e63640 20 client.4271.objecter shutdown clearing up homeless session... 2024-03-29T11:10:06.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:06.330+0000 7efec5e63640 10 client.4271.objecter successfully canceled tick 2024-03-29T11:10:06.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:06.330+0000 7efec5e63640 1 -- 172.21.15.18:0/4061674981 >> v1:172.21.15.70:6816/3101871562 conn(0x7efeac06e120 legacy=0x7efeac0705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:06.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:06.330+0000 7efec5e63640 10 monclient: shutdown 2024-03-29T11:10:06.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:06.330+0000 7efec5e63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:06.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:06.330+0000 7efec5e63640 1 -- 172.21.15.18:0/4061674981 >> v1:172.21.15.18:6790/0 conn(0x7efec0143d10 legacy=0x7efec01390c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:06.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:06.330+0000 7efec5e63640 1 -- 172.21.15.18:0/4061674981 shutdown_connections 2024-03-29T11:10:06.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:06.330+0000 7efec5e63640 1 -- 172.21.15.18:0/4061674981 >> 172.21.15.18:0/4061674981 conn(0x7efec0066a00 msgr2=0x7efec0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:06.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:06.330+0000 7efec5e63640 1 -- 172.21.15.18:0/4061674981 shutdown_connections 2024-03-29T11:10:06.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:06.330+0000 7efec5e63640 1 -- 172.21.15.18:0/4061674981 wait complete. 2024-03-29T11:10:06.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:06.331+0000 7efec5e63640 1 librados: shutdown 2024-03-29T11:10:06.325 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:10:13.132 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:10:13.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.315+0000 7fe4a0354640 10 monclient: get_monmap_and_config 2024-03-29T11:10:13.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.315+0000 7fe4a0354640 10 monclient: build_initial_monmap 2024-03-29T11:10:13.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.315+0000 7fe4a0354640 10 monclient: monmap: 2024-03-29T11:10:13.295 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:10:13.295 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:13.295 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:10:13.316052+0000 2024-03-29T11:10:13.295 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:10:13.316052+0000 2024-03-29T11:10:13.295 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:13.295 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:13.295 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:10:13.295 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:10:13.295 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:10:13.295 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:13.295 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.315+0000 7fe4a0354640 1 Processor -- start 2024-03-29T11:10:13.296 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.316+0000 7fe4a0354640 1 -- start start 2024-03-29T11:10:13.296 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.316+0000 7fe4a0354640 10 monclient: init 2024-03-29T11:10:13.298 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.317+0000 7fe4a0354640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:13.299 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.317+0000 7fe4a0354640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:10:13.299 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.317+0000 7fe4a0354640 10 monclient(hunting): picked mon.noname-b con 0x7fe4980a9750 addr v1:172.21.15.18:6790/0 2024-03-29T11:10:13.299 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.317+0000 7fe4a0354640 10 monclient(hunting): picked mon.noname-a con 0x7fe498149ac0 addr v1:172.21.15.18:6789/0 2024-03-29T11:10:13.299 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.317+0000 7fe4a0354640 10 monclient(hunting): picked mon.noname-c con 0x7fe49814fe50 addr v1:172.21.15.70:6789/0 2024-03-29T11:10:13.299 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.317+0000 7fe4a0354640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4980aa4a0 con 0x7fe498149ac0 2024-03-29T11:10:13.299 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.317+0000 7fe4a0354640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe49814a680 con 0x7fe49814fe50 2024-03-29T11:10:13.299 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.317+0000 7fe4a0354640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4980af290 con 0x7fe4980a9750 2024-03-29T11:10:13.299 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.317+0000 7fe4a0354640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:13.299 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.317+0000 7fe4a0354640 10 monclient(hunting): authenticate will time out at 839.589731s 2024-03-29T11:10:13.299 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.318+0000 7fe49e0f0640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fe4980a9750 0x7fe4981436f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:35190/0 (socket says 172.21.15.18:35190) 2024-03-29T11:10:13.299 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.318+0000 7fe49e0f0640 1 -- 172.21.15.18:0/2438517290 learned_addr learned my addr 172.21.15.18:0/2438517290 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:10:13.300 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.321+0000 7fe49d0ee640 1 -- 172.21.15.18:0/2438517290 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1231097040 0 0) 0x7fe4980af290 con 0x7fe4980a9750 2024-03-29T11:10:13.300 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.321+0000 7fe49d0ee640 10 monclient(hunting): _negotiate 2024-03-29T11:10:13.300 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.321+0000 7fe49d0ee640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:13.300 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.321+0000 7fe49d0ee640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:13.300 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.321+0000 7fe49d0ee640 10 monclient(hunting): my global_id is 4274 2024-03-29T11:10:13.301 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.321+0000 7fe49d0ee640 1 -- 172.21.15.18:0/2438517290 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe484003520 con 0x7fe4980a9750 2024-03-29T11:10:13.301 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.322+0000 7fe49d0ee640 1 -- 172.21.15.18:0/2438517290 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4158075601 0 0) 0x7fe484003520 con 0x7fe4980a9750 2024-03-29T11:10:13.301 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.322+0000 7fe49d0ee640 1 -- 172.21.15.18:0/2438517290 >> v1:172.21.15.70:6789/0 conn(0x7fe49814fe50 legacy=0x7fe498152240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:10:13.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.324+0000 7fe49d0ee640 1 -- 172.21.15.18:0/2438517290 >> v1:172.21.15.18:6789/0 conn(0x7fe498149ac0 legacy=0x7fe498149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:13.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.324+0000 7fe49d0ee640 10 monclient: _finish_hunting 0 2024-03-29T11:10:13.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.324+0000 7fe49d0ee640 1 monclient: found mon.noname-b 2024-03-29T11:10:13.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.324+0000 7fe49d0ee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:13.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.324+0000 7fe49d0ee640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:10:13.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.324+0000 7fe49d0ee640 1 -- 172.21.15.18:0/2438517290 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4980afb40 con 0x7fe4980a9750 2024-03-29T11:10:13.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.324+0000 7fe49d0ee640 10 monclient: _finish_auth 0 2024-03-29T11:10:13.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.325+0000 7fe49d0ee640 10 monclient: _check_auth_tickets 2024-03-29T11:10:13.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.325+0000 7fe49d0ee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:13.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.325+0000 7fe4a0354640 5 monclient: authenticate success, global_id 4274 2024-03-29T11:10:13.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.325+0000 7fe4a0354640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:10:13.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.325+0000 7fe49d0ee640 1 -- 172.21.15.18:0/2438517290 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fe494001690 con 0x7fe4980a9750 2024-03-29T11:10:13.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.325+0000 7fe49d0ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:13.305 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.325+0000 7fe49d0ee640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:10:13.305 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.325+0000 7fe49d0ee640 10 monclient: dump: 2024-03-29T11:10:13.305 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:10:13.305 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:13.305 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:13.305 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:13.305 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:13.305 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:13.305 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:10:13.305 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:10:13.305 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:10:13.305 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:13.305 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.325+0000 7fe4a0354640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:10:13.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.330+0000 7fe49d0ee640 1 -- 172.21.15.18:0/2438517290 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe494001860 con 0x7fe4980a9750 2024-03-29T11:10:13.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.330+0000 7fe49d0ee640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:13.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.330+0000 7fe4a0354640 10 monclient: get_monmap_and_config success 2024-03-29T11:10:13.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.330+0000 7fe4a0354640 4 set_mon_vals no callback set 2024-03-29T11:10:13.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.330+0000 7fe49d0ee640 1 -- 172.21.15.18:0/2438517290 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fe494001cb0 con 0x7fe4980a9750 2024-03-29T11:10:13.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.331+0000 7fe49d0ee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:13.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.331+0000 7fe49d0ee640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:10:13.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.331+0000 7fe49d0ee640 10 monclient: dump: 2024-03-29T11:10:13.310 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:10:13.310 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:13.310 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:13.310 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:13.310 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:13.311 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:13.311 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:10:13.311 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:10:13.311 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:10:13.311 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:13.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.331+0000 7fe4a0354640 10 monclient: shutdown 2024-03-29T11:10:13.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.331+0000 7fe4a0354640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:13.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.331+0000 7fe4a0354640 1 -- 172.21.15.18:0/2438517290 >> v1:172.21.15.18:6790/0 conn(0x7fe4980a9750 legacy=0x7fe4981436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:13.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.331+0000 7fe4a0354640 1 -- 172.21.15.18:0/2438517290 shutdown_connections 2024-03-29T11:10:13.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.331+0000 7fe4a0354640 1 -- 172.21.15.18:0/2438517290 >> 172.21.15.18:0/2438517290 conn(0x7fe4980b35e0 msgr2=0x7fe4980b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:13.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.332+0000 7fe4a0354640 1 -- 172.21.15.18:0/2438517290 shutdown_connections 2024-03-29T11:10:13.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.332+0000 7fe4a0354640 1 -- 172.21.15.18:0/2438517290 wait complete. 2024-03-29T11:10:13.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.333+0000 7fe4a0354640 10 monclient: build_initial_monmap 2024-03-29T11:10:13.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.333+0000 7fe4a0354640 10 monclient: monmap: 2024-03-29T11:10:13.312 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:10:13.312 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:10:13.312 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:10:13.312 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:10:13.312 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:13.312 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:13.313 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:10:13.313 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:10:13.313 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:10:13.313 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:13.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.333+0000 7fe4a0354640 1 librados: starting msgr at 2024-03-29T11:10:13.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.333+0000 7fe4a0354640 1 librados: starting objecter 2024-03-29T11:10:13.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.333+0000 7fe4a0354640 1 Processor -- start 2024-03-29T11:10:13.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.333+0000 7fe4a0354640 1 -- start start 2024-03-29T11:10:13.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.333+0000 7fe4a0354640 1 librados: setting wanted keys 2024-03-29T11:10:13.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.333+0000 7fe4a0354640 1 librados: calling monclient init 2024-03-29T11:10:13.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.333+0000 7fe4a0354640 10 monclient: init 2024-03-29T11:10:13.314 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.334+0000 7fe4a0354640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:13.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.334+0000 7fe4a0354640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:10:13.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.334+0000 7fe4a0354640 10 monclient(hunting): picked mon.noname-b con 0x7fe4980a9750 addr v1:172.21.15.70:6789/0 2024-03-29T11:10:13.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.334+0000 7fe4a0354640 10 monclient(hunting): picked mon.noname-a con 0x7fe498149ac0 addr v1:172.21.15.18:6789/0 2024-03-29T11:10:13.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.336+0000 7fe4a0354640 10 monclient(hunting): picked mon.noname-c con 0x7fe49814fe50 addr v1:172.21.15.18:6790/0 2024-03-29T11:10:13.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.336+0000 7fe4a0354640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe49813d850 con 0x7fe498149ac0 2024-03-29T11:10:13.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.336+0000 7fe4a0354640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe49813c570 con 0x7fe4980a9750 2024-03-29T11:10:13.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.336+0000 7fe4a0354640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe49813c700 con 0x7fe49814fe50 2024-03-29T11:10:13.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.336+0000 7fe4a0354640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:13.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.336+0000 7fe4a0354640 10 monclient(hunting): authenticate will time out at 839.608277s 2024-03-29T11:10:13.316 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.338+0000 7fe49d8ef640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fe498149ac0 0x7fe4981e3570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:40055/0 (socket says 172.21.15.18:40055) 2024-03-29T11:10:13.316 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.338+0000 7fe49d8ef640 1 -- 172.21.15.18:0/2694081468 learned_addr learned my addr 172.21.15.18:0/2694081468 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:10:13.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.339+0000 7fe48effd640 10 client.?.objecter ms_handle_connect 0x7fe498149ac0 2024-03-29T11:10:13.317 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.339+0000 7fe48effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:13.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.341+0000 7fe48effd640 10 client.?.objecter ms_handle_connect 0x7fe49814fe50 2024-03-29T11:10:13.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.341+0000 7fe48effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:13.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.342+0000 7fe48effd640 10 client.?.objecter ms_handle_connect 0x7fe4980a9750 2024-03-29T11:10:13.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.342+0000 7fe48effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:13.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.342+0000 7fe48effd640 1 -- 172.21.15.18:0/2694081468 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2237205988 0 0) 0x7fe49813d850 con 0x7fe498149ac0 2024-03-29T11:10:13.321 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.342+0000 7fe48effd640 10 monclient(hunting): _negotiate 2024-03-29T11:10:13.321 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.342+0000 7fe48effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:13.321 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.342+0000 7fe48effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:13.321 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.342+0000 7fe48effd640 10 monclient(hunting): my global_id is 4278 2024-03-29T11:10:13.321 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.342+0000 7fe48effd640 1 -- 172.21.15.18:0/2694081468 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe47c004eb0 con 0x7fe498149ac0 2024-03-29T11:10:13.323 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.343+0000 7fe48effd640 1 -- 172.21.15.18:0/2694081468 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4007864001 0 0) 0x7fe49813c570 con 0x7fe4980a9750 2024-03-29T11:10:13.323 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.343+0000 7fe48effd640 10 monclient(hunting): _negotiate 2024-03-29T11:10:13.323 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.343+0000 7fe48effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:13.323 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.343+0000 7fe48effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:13.323 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.343+0000 7fe48effd640 10 monclient(hunting): my global_id is 4270 2024-03-29T11:10:13.323 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.343+0000 7fe48effd640 1 -- 172.21.15.18:0/2694081468 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe49813d850 con 0x7fe4980a9750 2024-03-29T11:10:13.323 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.343+0000 7fe48effd640 1 -- 172.21.15.18:0/2694081468 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2084573216 0 0) 0x7fe47c004eb0 con 0x7fe498149ac0 2024-03-29T11:10:13.324 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.343+0000 7fe48effd640 1 -- 172.21.15.18:0/2694081468 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe49813c570 con 0x7fe498149ac0 2024-03-29T11:10:13.324 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.343+0000 7fe48effd640 1 -- 172.21.15.18:0/2694081468 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fe488001a10 con 0x7fe498149ac0 2024-03-29T11:10:13.324 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.343+0000 7fe48effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:13.324 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.343+0000 7fe48effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:10:13.324 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.343+0000 7fe48effd640 10 monclient(hunting): dump: 2024-03-29T11:10:13.324 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:10:13.324 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:13.324 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:13.324 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:13.324 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:13.324 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:13.324 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:10:13.324 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:10:13.324 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:10:13.324 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:13.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.343+0000 7fe48effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:10:13.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.343+0000 7fe48effd640 1 -- 172.21.15.18:0/2694081468 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2940796841 0 0) 0x7fe49813c570 con 0x7fe498149ac0 2024-03-29T11:10:13.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.343+0000 7fe48effd640 1 -- 172.21.15.18:0/2694081468 >> v1:172.21.15.18:6790/0 conn(0x7fe49814fe50 legacy=0x7fe4981e44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:13.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.343+0000 7fe48effd640 1 -- 172.21.15.18:0/2694081468 >> v1:172.21.15.70:6789/0 conn(0x7fe4980a9750 legacy=0x7fe49813d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:13.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.343+0000 7fe48effd640 10 monclient: _finish_hunting 0 2024-03-29T11:10:13.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.344+0000 7fe48effd640 1 monclient: found mon.a 2024-03-29T11:10:13.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.344+0000 7fe48effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:13.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.344+0000 7fe48effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:10:13.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.344+0000 7fe48effd640 1 -- 172.21.15.18:0/2694081468 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4981401b0 con 0x7fe498149ac0 2024-03-29T11:10:13.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.344+0000 7fe48effd640 10 monclient: _finish_auth 0 2024-03-29T11:10:13.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.344+0000 7fe48effd640 10 monclient: _check_auth_tickets 2024-03-29T11:10:13.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.344+0000 7fe48effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:13.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.344+0000 7fe4a0354640 5 monclient: authenticate success, global_id 4278 2024-03-29T11:10:13.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.344+0000 7fe4a0354640 10 monclient: _renew_subs 2024-03-29T11:10:13.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.344+0000 7fe4a0354640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:10:13.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.344+0000 7fe4a0354640 1 -- 172.21.15.18:0/2694081468 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe498140460 con 0x7fe498149ac0 2024-03-29T11:10:13.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.344+0000 7fe4a0354640 10 client.4278.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:10:13.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.345+0000 7fe4a0354640 10 monclient: _renew_subs 2024-03-29T11:10:13.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.345+0000 7fe4a0354640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:10:13.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.345+0000 7fe4a0354640 1 -- 172.21.15.18:0/2694081468 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4981e3e60 con 0x7fe498149ac0 2024-03-29T11:10:13.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.345+0000 7fe4a0354640 1 librados: init done 2024-03-29T11:10:13.328 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.346+0000 7fe48effd640 1 -- 172.21.15.18:0/2694081468 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe488001b70 con 0x7fe498149ac0 2024-03-29T11:10:13.328 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.346+0000 7fe48effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:13.328 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.346+0000 7fe48effd640 1 -- 172.21.15.18:0/2694081468 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fe488003a60 con 0x7fe498149ac0 2024-03-29T11:10:13.328 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.346+0000 7fe48effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:13.328 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.346+0000 7fe48effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:10:13.328 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.346+0000 7fe48effd640 10 monclient: dump: 2024-03-29T11:10:13.328 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:10:13.328 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:13.328 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:13.329 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:13.329 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:13.329 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:13.329 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:10:13.329 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:10:13.329 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:10:13.329 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:13.329 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.346+0000 7fe49c8ed640 4 set_mon_vals no callback set 2024-03-29T11:10:13.329 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.347+0000 7fe4a0354640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:13.329 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.347+0000 7fe4a0354640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:13.329 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.347+0000 7fe4a0354640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:10:13.329 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.347+0000 7fe4a0354640 1 -- 172.21.15.18:0/2694081468 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe480005080 con 0x7fe498149ac0 2024-03-29T11:10:13.329 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.348+0000 7fe48effd640 1 -- 172.21.15.18:0/2694081468 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7fe488003bc0 con 0x7fe498149ac0 2024-03-29T11:10:13.329 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.349+0000 7fe49e0f0640 10 monclient: get_auth_request con 0x7fe47c06dd60 auth_method 0 2024-03-29T11:10:13.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.349+0000 7fe48effd640 1 -- 172.21.15.18:0/2694081468 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7fe488084a90 con 0x7fe498149ac0 2024-03-29T11:10:13.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.349+0000 7fe48effd640 10 client.4278.objecter ms_dispatch 0x7fe498000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:10:13.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.349+0000 7fe48effd640 3 client.4278.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:10:13.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.350+0000 7fe48effd640 3 client.4278.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:10:13.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.350+0000 7fe48effd640 20 client.4278.objecter dump_active .. 0 homeless 2024-03-29T11:10:13.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.350+0000 7fe48effd640 10 client.4278.objecter ms_handle_connect 0x7fe47c06dd60 2024-03-29T11:10:13.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.355+0000 7fe48effd640 1 -- 172.21.15.18:0/2694081468 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe488050370 con 0x7fe498149ac0 2024-03-29T11:10:13.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.355+0000 7fe48effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:13.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.355+0000 7fe48effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:10:13.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.680+0000 7fe4a0354640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:13.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.680+0000 7fe4a0354640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:13.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.680+0000 7fe4a0354640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:10:13.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.680+0000 7fe4a0354640 1 -- 172.21.15.18:0/2694081468 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe480005330 con 0x7fe498149ac0 2024-03-29T11:10:13.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.685+0000 7fe48effd640 1 -- 172.21.15.18:0/2694081468 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7fe4880522a0 con 0x7fe498149ac0 2024-03-29T11:10:13.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.685+0000 7fe48effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:13.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.685+0000 7fe48effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:10:13.663 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:10:13.663 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:10:13.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.688+0000 7fe4a0354640 10 librados: watch_flush enter 2024-03-29T11:10:13.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.688+0000 7fe4a0354640 10 librados: watch_flush exit 2024-03-29T11:10:13.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.688+0000 7fe4a0354640 20 client.4278.objecter shutdown clearing up homeless session... 2024-03-29T11:10:13.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.688+0000 7fe4a0354640 10 client.4278.objecter successfully canceled tick 2024-03-29T11:10:13.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.688+0000 7fe4a0354640 1 -- 172.21.15.18:0/2694081468 >> v1:172.21.15.70:6816/3101871562 conn(0x7fe47c06dd60 legacy=0x7fe47c070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:13.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.688+0000 7fe4a0354640 10 monclient: shutdown 2024-03-29T11:10:13.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.688+0000 7fe4a0354640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:13.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.688+0000 7fe4a0354640 1 -- 172.21.15.18:0/2694081468 >> v1:172.21.15.18:6789/0 conn(0x7fe498149ac0 legacy=0x7fe4981e3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:13.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.689+0000 7fe4a0354640 1 -- 172.21.15.18:0/2694081468 shutdown_connections 2024-03-29T11:10:13.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.689+0000 7fe4a0354640 1 -- 172.21.15.18:0/2694081468 >> 172.21.15.18:0/2694081468 conn(0x7fe4980b35e0 msgr2=0x7fe4981434c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:13.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.689+0000 7fe4a0354640 1 -- 172.21.15.18:0/2694081468 shutdown_connections 2024-03-29T11:10:13.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.690+0000 7fe4a0354640 1 -- 172.21.15.18:0/2694081468 wait complete. 2024-03-29T11:10:13.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:13.690+0000 7fe4a0354640 1 librados: shutdown 2024-03-29T11:10:13.686 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:10:20.493 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:10:20.652 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.674+0000 7f239cc45640 10 monclient: get_monmap_and_config 2024-03-29T11:10:20.652 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.674+0000 7f239cc45640 10 monclient: build_initial_monmap 2024-03-29T11:10:20.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.674+0000 7f239cc45640 10 monclient: monmap: 2024-03-29T11:10:20.653 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:10:20.653 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:20.653 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:10:20.675059+0000 2024-03-29T11:10:20.653 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:10:20.675059+0000 2024-03-29T11:10:20.653 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:20.653 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:20.653 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:10:20.653 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:10:20.653 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:10:20.653 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:20.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.675+0000 7f239cc45640 1 Processor -- start 2024-03-29T11:10:20.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.675+0000 7f239cc45640 1 -- start start 2024-03-29T11:10:20.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.675+0000 7f239cc45640 10 monclient: init 2024-03-29T11:10:20.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.676+0000 7f239cc45640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:20.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.676+0000 7f239cc45640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:10:20.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.676+0000 7f239cc45640 10 monclient(hunting): picked mon.noname-a con 0x7f23980b0770 addr v1:172.21.15.18:6789/0 2024-03-29T11:10:20.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.676+0000 7f239cc45640 10 monclient(hunting): picked mon.noname-b con 0x7f239814a700 addr v1:172.21.15.18:6790/0 2024-03-29T11:10:20.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.676+0000 7f239cc45640 10 monclient(hunting): picked mon.noname-c con 0x7f23980ac5a0 addr v1:172.21.15.70:6789/0 2024-03-29T11:10:20.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.676+0000 7f239cc45640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23980aba70 con 0x7f23980b0770 2024-03-29T11:10:20.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.676+0000 7f239cc45640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23980aecb0 con 0x7f23980ac5a0 2024-03-29T11:10:20.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.676+0000 7f239cc45640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23980af150 con 0x7f239814a700 2024-03-29T11:10:20.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.676+0000 7f239cc45640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:20.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.676+0000 7f239cc45640 10 monclient(hunting): authenticate will time out at 846.948433s 2024-03-29T11:10:20.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.676+0000 7f239659c640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f23980b0770 0x7f2398149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:33100/0 (socket says 172.21.15.18:33100) 2024-03-29T11:10:20.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.676+0000 7f2395d9b640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f239814a700 0x7f23980abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:40066/0 (socket says 172.21.15.18:40066) 2024-03-29T11:10:20.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.676+0000 7f239659c640 1 -- 172.21.15.18:0/3854543097 learned_addr learned my addr 172.21.15.18:0/3854543097 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:10:20.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.677+0000 7f239559a640 1 -- 172.21.15.18:0/3854543097 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1836340550 0 0) 0x7f23980aba70 con 0x7f23980b0770 2024-03-29T11:10:20.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.677+0000 7f239559a640 10 monclient(hunting): _negotiate 2024-03-29T11:10:20.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.677+0000 7f239559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:20.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.677+0000 7f239559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:20.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.677+0000 7f239559a640 10 monclient(hunting): my global_id is 4281 2024-03-29T11:10:20.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.677+0000 7f239559a640 1 -- 172.21.15.18:0/3854543097 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2380003520 con 0x7f23980b0770 2024-03-29T11:10:20.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.677+0000 7f239559a640 1 -- 172.21.15.18:0/3854543097 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1575219569 0 0) 0x7f23980af150 con 0x7f239814a700 2024-03-29T11:10:20.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.677+0000 7f239559a640 10 monclient(hunting): _negotiate 2024-03-29T11:10:20.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.677+0000 7f239559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:20.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.677+0000 7f239559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:20.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.677+0000 7f239559a640 10 monclient(hunting): my global_id is 4280 2024-03-29T11:10:20.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.677+0000 7f239559a640 1 -- 172.21.15.18:0/3854543097 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2380003840 con 0x7f239814a700 2024-03-29T11:10:20.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.678+0000 7f239559a640 1 -- 172.21.15.18:0/3854543097 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442626747 0 0) 0x7f23980aecb0 con 0x7f23980ac5a0 2024-03-29T11:10:20.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.678+0000 7f239559a640 10 monclient(hunting): _negotiate 2024-03-29T11:10:20.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.678+0000 7f239559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:20.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.678+0000 7f239559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:20.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.678+0000 7f239559a640 10 monclient(hunting): my global_id is 4273 2024-03-29T11:10:20.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.678+0000 7f239559a640 1 -- 172.21.15.18:0/3854543097 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23980af150 con 0x7f23980ac5a0 2024-03-29T11:10:20.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.678+0000 7f239559a640 1 -- 172.21.15.18:0/3854543097 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2392489049 0 0) 0x7f2380003520 con 0x7f23980b0770 2024-03-29T11:10:20.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.678+0000 7f239559a640 1 -- 172.21.15.18:0/3854543097 >> v1:172.21.15.18:6790/0 conn(0x7f239814a700 legacy=0x7f23980abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:20.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.678+0000 7f239559a640 1 -- 172.21.15.18:0/3854543097 >> v1:172.21.15.70:6789/0 conn(0x7f23980ac5a0 legacy=0x7f2398151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:20.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.678+0000 7f239559a640 10 monclient: _finish_hunting 0 2024-03-29T11:10:20.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.678+0000 7f239559a640 1 monclient: found mon.noname-a 2024-03-29T11:10:20.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.678+0000 7f239559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:20.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.678+0000 7f239559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-29T11:10:20.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.678+0000 7f239559a640 1 -- 172.21.15.18:0/3854543097 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23980ac9d0 con 0x7f23980b0770 2024-03-29T11:10:20.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.678+0000 7f239559a640 10 monclient: _finish_auth 0 2024-03-29T11:10:20.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.678+0000 7f239559a640 10 monclient: _check_auth_tickets 2024-03-29T11:10:20.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.678+0000 7f239559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:20.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.678+0000 7f239559a640 1 -- 172.21.15.18:0/3854543097 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f23900015a0 con 0x7f23980b0770 2024-03-29T11:10:20.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.678+0000 7f239559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:20.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.678+0000 7f239559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:10:20.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.678+0000 7f239559a640 10 monclient: dump: 2024-03-29T11:10:20.662 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:10:20.662 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:20.662 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:20.662 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:20.662 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:20.662 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:20.662 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:10:20.663 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:10:20.663 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:10:20.663 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:20.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.678+0000 7f239cc45640 5 monclient: authenticate success, global_id 4281 2024-03-29T11:10:20.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.678+0000 7f239cc45640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:10:20.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.678+0000 7f239559a640 1 -- 172.21.15.18:0/3854543097 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2390001780 con 0x7f23980b0770 2024-03-29T11:10:20.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.678+0000 7f239559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:20.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.678+0000 7f239cc45640 10 monclient: get_monmap_and_config success 2024-03-29T11:10:20.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.678+0000 7f239cc45640 4 set_mon_vals no callback set 2024-03-29T11:10:20.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.679+0000 7f239cc45640 10 monclient: shutdown 2024-03-29T11:10:20.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.679+0000 7f239cc45640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:20.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.679+0000 7f239cc45640 1 -- 172.21.15.18:0/3854543097 >> v1:172.21.15.18:6789/0 conn(0x7f23980b0770 legacy=0x7f2398149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:20.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.679+0000 7f239559a640 1 -- 172.21.15.18:0/3854543097 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f23900015a0 con 0x7f23980b0770 2024-03-29T11:10:20.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.679+0000 7f239559a640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:10:20.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.680+0000 7f239cc45640 1 -- 172.21.15.18:0/3854543097 shutdown_connections 2024-03-29T11:10:20.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.680+0000 7f239cc45640 1 -- 172.21.15.18:0/3854543097 >> 172.21.15.18:0/3854543097 conn(0x7f239806b660 msgr2=0x7f239806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:20.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.680+0000 7f239cc45640 1 -- 172.21.15.18:0/3854543097 shutdown_connections 2024-03-29T11:10:20.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.680+0000 7f239cc45640 1 -- 172.21.15.18:0/3854543097 wait complete. 2024-03-29T11:10:20.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.682+0000 7f239cc45640 10 monclient: build_initial_monmap 2024-03-29T11:10:20.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.682+0000 7f239cc45640 10 monclient: monmap: 2024-03-29T11:10:20.672 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:10:20.673 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:10:20.673 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:10:20.673 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:10:20.673 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:20.673 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:20.673 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:10:20.673 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:10:20.673 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:10:20.673 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:20.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.682+0000 7f239cc45640 1 librados: starting msgr at 2024-03-29T11:10:20.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.682+0000 7f239cc45640 1 librados: starting objecter 2024-03-29T11:10:20.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.682+0000 7f239cc45640 1 Processor -- start 2024-03-29T11:10:20.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.682+0000 7f239cc45640 1 -- start start 2024-03-29T11:10:20.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.683+0000 7f239cc45640 1 librados: setting wanted keys 2024-03-29T11:10:20.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.683+0000 7f239cc45640 1 librados: calling monclient init 2024-03-29T11:10:20.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.683+0000 7f239cc45640 10 monclient: init 2024-03-29T11:10:20.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.683+0000 7f239cc45640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:20.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.683+0000 7f239cc45640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:10:20.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.683+0000 7f239cc45640 10 monclient(hunting): picked mon.noname-c con 0x7f23980ac5a0 addr v1:172.21.15.18:6790/0 2024-03-29T11:10:20.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.684+0000 7f239cc45640 10 monclient(hunting): picked mon.noname-a con 0x7f23980b0770 addr v1:172.21.15.18:6789/0 2024-03-29T11:10:20.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.684+0000 7f239cc45640 10 monclient(hunting): picked mon.noname-b con 0x7f239814a700 addr v1:172.21.15.70:6789/0 2024-03-29T11:10:20.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.684+0000 7f239659c640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f23980ac5a0 0x7f2398126760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:40234/0 (socket says 172.21.15.18:40234) 2024-03-29T11:10:20.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.684+0000 7f239659c640 1 -- 172.21.15.18:0/3127831941 learned_addr learned my addr 172.21.15.18:0/3127831941 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:10:20.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.684+0000 7f2386ffd640 10 client.?.objecter ms_handle_connect 0x7f23980ac5a0 2024-03-29T11:10:20.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.684+0000 7f2386ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:20.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.684+0000 7f2386ffd640 10 client.?.objecter ms_handle_connect 0x7f239814a700 2024-03-29T11:10:20.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.684+0000 7f2386ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:20.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.685+0000 7f2386ffd640 10 client.?.objecter ms_handle_connect 0x7f23980b0770 2024-03-29T11:10:20.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.685+0000 7f2386ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:20.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.685+0000 7f239cc45640 1 -- 172.21.15.18:0/3127831941 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23981239d0 con 0x7f23980b0770 2024-03-29T11:10:20.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.685+0000 7f239cc45640 1 -- 172.21.15.18:0/3127831941 --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2398145b00 con 0x7f239814a700 2024-03-29T11:10:20.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.685+0000 7f239cc45640 1 -- 172.21.15.18:0/3127831941 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2398125cf0 con 0x7f23980ac5a0 2024-03-29T11:10:20.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.685+0000 7f239cc45640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:20.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.685+0000 7f239cc45640 10 monclient(hunting): authenticate will time out at 846.957287s 2024-03-29T11:10:20.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.685+0000 7f2386ffd640 1 -- 172.21.15.18:0/3127831941 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4001430332 0 0) 0x7f2398145b00 con 0x7f239814a700 2024-03-29T11:10:20.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.685+0000 7f2386ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:10:20.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.685+0000 7f2386ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:20.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.685+0000 7f2386ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:20.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.685+0000 7f2386ffd640 10 monclient(hunting): my global_id is 4276 2024-03-29T11:10:20.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.685+0000 7f2386ffd640 1 -- 172.21.15.18:0/3127831941 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2378004eb0 con 0x7f239814a700 2024-03-29T11:10:20.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.685+0000 7f2386ffd640 1 -- 172.21.15.18:0/3127831941 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 433169181 0 0) 0x7f23981239d0 con 0x7f23980b0770 2024-03-29T11:10:20.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.685+0000 7f2386ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:10:20.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.685+0000 7f2386ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:20.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.685+0000 7f2386ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:20.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.685+0000 7f2386ffd640 10 monclient(hunting): my global_id is 4284 2024-03-29T11:10:20.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.685+0000 7f2386ffd640 1 -- 172.21.15.18:0/3127831941 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23780051d0 con 0x7f23980b0770 2024-03-29T11:10:20.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.686+0000 7f2386ffd640 1 -- 172.21.15.18:0/3127831941 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2469906239 0 0) 0x7f2398125cf0 con 0x7f23980ac5a0 2024-03-29T11:10:20.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.686+0000 7f2386ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:10:20.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.686+0000 7f2386ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:20.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.686+0000 7f2386ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:20.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.686+0000 7f2386ffd640 10 monclient(hunting): my global_id is 4283 2024-03-29T11:10:20.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.686+0000 7f2386ffd640 1 -- 172.21.15.18:0/3127831941 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23981239d0 con 0x7f23980ac5a0 2024-03-29T11:10:20.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.686+0000 7f2386ffd640 1 -- 172.21.15.18:0/3127831941 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2024712421 0 0) 0x7f2378004eb0 con 0x7f239814a700 2024-03-29T11:10:20.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.686+0000 7f2386ffd640 1 -- 172.21.15.18:0/3127831941 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2398125cf0 con 0x7f239814a700 2024-03-29T11:10:20.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.686+0000 7f2386ffd640 1 -- 172.21.15.18:0/3127831941 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f238c001c70 con 0x7f239814a700 2024-03-29T11:10:20.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.686+0000 7f2386ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:20.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.686+0000 7f2386ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:10:20.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.686+0000 7f2386ffd640 10 monclient(hunting): dump: 2024-03-29T11:10:20.677 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:10:20.677 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:20.677 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:20.677 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:20.677 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:20.677 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:20.677 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:10:20.677 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:10:20.677 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:10:20.677 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:20.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.686+0000 7f2386ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:10:20.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.686+0000 7f2386ffd640 1 -- 172.21.15.18:0/3127831941 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 301231955 0 0) 0x7f23780051d0 con 0x7f23980b0770 2024-03-29T11:10:20.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.686+0000 7f2386ffd640 1 -- 172.21.15.18:0/3127831941 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2378004eb0 con 0x7f23980b0770 2024-03-29T11:10:20.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.686+0000 7f2386ffd640 1 -- 172.21.15.18:0/3127831941 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f2388001770 con 0x7f23980b0770 2024-03-29T11:10:20.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.686+0000 7f2386ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:20.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.686+0000 7f2386ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:10:20.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.686+0000 7f2386ffd640 10 monclient(hunting): dump: 2024-03-29T11:10:20.678 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:10:20.678 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:20.678 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:20.678 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:20.678 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:20.678 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:20.678 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:10:20.678 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:10:20.678 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:10:20.678 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:20.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.686+0000 7f2386ffd640 1 -- 172.21.15.18:0/3127831941 <== mon.1 v1:172.21.15.70:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2121954435 0 0) 0x7f2398125cf0 con 0x7f239814a700 2024-03-29T11:10:20.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.686+0000 7f2386ffd640 1 -- 172.21.15.18:0/3127831941 >> v1:172.21.15.18:6790/0 conn(0x7f23980ac5a0 legacy=0x7f2398126760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:20.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.686+0000 7f2386ffd640 1 -- 172.21.15.18:0/3127831941 >> v1:172.21.15.18:6789/0 conn(0x7f23980b0770 legacy=0x7f23981db870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:20.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.686+0000 7f2386ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:10:20.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.686+0000 7f2386ffd640 1 monclient: found mon.b 2024-03-29T11:10:20.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.686+0000 7f2386ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:20.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.686+0000 7f2386ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:10:20.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.686+0000 7f2386ffd640 1 -- 172.21.15.18:0/3127831941 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2398126e70 con 0x7f239814a700 2024-03-29T11:10:20.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.686+0000 7f2386ffd640 10 monclient: _finish_auth 0 2024-03-29T11:10:20.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.686+0000 7f2386ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:10:20.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.686+0000 7f2386ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:20.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.687+0000 7f2386ffd640 1 -- 172.21.15.18:0/3127831941 <== mon.1 v1:172.21.15.70:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f238c003990 con 0x7f239814a700 2024-03-29T11:10:20.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.687+0000 7f2386ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:20.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.687+0000 7f2386ffd640 1 -- 172.21.15.18:0/3127831941 <== mon.1 v1:172.21.15.70:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f238c003d20 con 0x7f239814a700 2024-03-29T11:10:20.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.687+0000 7f2386ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:20.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.687+0000 7f2386ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:10:20.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.687+0000 7f2386ffd640 10 monclient: dump: 2024-03-29T11:10:20.680 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:10:20.680 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:20.680 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:20.680 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:20.680 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:20.680 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:20.680 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:10:20.680 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:10:20.680 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:10:20.680 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:20.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.687+0000 7f2394d99640 4 set_mon_vals no callback set 2024-03-29T11:10:20.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.688+0000 7f239cc45640 5 monclient: authenticate success, global_id 4276 2024-03-29T11:10:20.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.688+0000 7f239cc45640 10 monclient: _renew_subs 2024-03-29T11:10:20.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.688+0000 7f239cc45640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:10:20.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.688+0000 7f239cc45640 1 -- 172.21.15.18:0/3127831941 --> v1:172.21.15.70:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23981dbf80 con 0x7f239814a700 2024-03-29T11:10:20.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.688+0000 7f239cc45640 10 client.4276.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:10:20.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.688+0000 7f239cc45640 10 monclient: _renew_subs 2024-03-29T11:10:20.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.688+0000 7f239cc45640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:10:20.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.688+0000 7f239cc45640 1 -- 172.21.15.18:0/3127831941 --> v1:172.21.15.70:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23981e7720 con 0x7f239814a700 2024-03-29T11:10:20.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.688+0000 7f239cc45640 1 librados: init done 2024-03-29T11:10:20.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.690+0000 7f239cc45640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:20.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.690+0000 7f239cc45640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:20.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.690+0000 7f239cc45640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:10:20.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.690+0000 7f239cc45640 1 -- 172.21.15.18:0/3127831941 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23980ab560 con 0x7f239814a700 2024-03-29T11:10:20.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.690+0000 7f2386ffd640 1 -- 172.21.15.18:0/3127831941 <== mon.1 v1:172.21.15.70:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f238c003e80 con 0x7f239814a700 2024-03-29T11:10:20.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.691+0000 7f2386ffd640 1 -- 172.21.15.18:0/3127831941 <== mon.1 v1:172.21.15.70:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f238c0538e0 con 0x7f239814a700 2024-03-29T11:10:20.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.691+0000 7f2386ffd640 10 client.4276.objecter ms_dispatch 0x7f2398000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:10:20.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.691+0000 7f2386ffd640 3 client.4276.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:10:20.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.691+0000 7f2386ffd640 3 client.4276.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:10:20.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.691+0000 7f2386ffd640 20 client.4276.objecter dump_active .. 0 homeless 2024-03-29T11:10:20.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.691+0000 7f239659c640 10 monclient: get_auth_request con 0x7f237806e140 auth_method 0 2024-03-29T11:10:20.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.692+0000 7f2386ffd640 10 client.4276.objecter ms_handle_connect 0x7f237806e140 2024-03-29T11:10:20.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.699+0000 7f2386ffd640 1 -- 172.21.15.18:0/3127831941 <== mon.1 v1:172.21.15.70:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f238c04fb80 con 0x7f239814a700 2024-03-29T11:10:20.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.699+0000 7f2386ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:20.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:20.699+0000 7f2386ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:10:21.054 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:21.076+0000 7f239cc45640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:21.055 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:21.076+0000 7f239cc45640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:21.055 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:21.076+0000 7f239cc45640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:10:21.055 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:21.076+0000 7f239cc45640 1 -- 172.21.15.18:0/3127831941 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2398128170 con 0x7f239814a700 2024-03-29T11:10:21.055 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:21.077+0000 7f2386ffd640 1 -- 172.21.15.18:0/3127831941 <== mon.1 v1:172.21.15.70:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f238c04fd20 con 0x7f239814a700 2024-03-29T11:10:21.055 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:21.077+0000 7f2386ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:21.056 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:21.077+0000 7f2386ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:10:21.056 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:10:21.056 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:10:21.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:21.080+0000 7f239cc45640 10 librados: watch_flush enter 2024-03-29T11:10:21.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:21.081+0000 7f239cc45640 10 librados: watch_flush exit 2024-03-29T11:10:21.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:21.081+0000 7f239cc45640 20 client.4276.objecter shutdown clearing up homeless session... 2024-03-29T11:10:21.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:21.081+0000 7f239cc45640 10 client.4276.objecter successfully canceled tick 2024-03-29T11:10:21.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:21.081+0000 7f239cc45640 1 -- 172.21.15.18:0/3127831941 >> v1:172.21.15.70:6816/3101871562 conn(0x7f237806e140 legacy=0x7f2378070600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:21.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:21.081+0000 7f239cc45640 10 monclient: shutdown 2024-03-29T11:10:21.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:21.081+0000 7f239cc45640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:21.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:21.081+0000 7f239cc45640 1 -- 172.21.15.18:0/3127831941 >> v1:172.21.15.70:6789/0 conn(0x7f239814a700 legacy=0x7f23981e6000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:21.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:21.081+0000 7f239cc45640 1 -- 172.21.15.18:0/3127831941 shutdown_connections 2024-03-29T11:10:21.060 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:21.081+0000 7f239cc45640 1 -- 172.21.15.18:0/3127831941 >> 172.21.15.18:0/3127831941 conn(0x7f239806b660 msgr2=0x7f23980b3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:21.060 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:21.081+0000 7f239cc45640 1 -- 172.21.15.18:0/3127831941 shutdown_connections 2024-03-29T11:10:21.060 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:21.081+0000 7f239cc45640 1 -- 172.21.15.18:0/3127831941 wait complete. 2024-03-29T11:10:21.060 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:21.082+0000 7f239cc45640 1 librados: shutdown 2024-03-29T11:10:21.073 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:10:27.880 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:10:28.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.036+0000 7f338359e640 10 monclient: get_monmap_and_config 2024-03-29T11:10:28.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.036+0000 7f338359e640 10 monclient: build_initial_monmap 2024-03-29T11:10:28.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.036+0000 7f338359e640 10 monclient: monmap: 2024-03-29T11:10:28.015 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:10:28.015 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:28.015 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:10:28.037179+0000 2024-03-29T11:10:28.015 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:10:28.037179+0000 2024-03-29T11:10:28.015 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:28.015 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:28.015 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:10:28.015 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:10:28.015 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:10:28.015 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:28.016 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.037+0000 7f338359e640 1 Processor -- start 2024-03-29T11:10:28.016 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.037+0000 7f338359e640 1 -- start start 2024-03-29T11:10:28.016 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.037+0000 7f338359e640 10 monclient: init 2024-03-29T11:10:28.016 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.038+0000 7f338359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:28.016 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.038+0000 7f338359e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:10:28.017 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.038+0000 7f338359e640 10 monclient(hunting): picked mon.noname-b con 0x7f337c006ad0 addr v1:172.21.15.18:6790/0 2024-03-29T11:10:28.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.039+0000 7f338259c640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f337c006ad0 0x7f337c0c4a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:42288/0 (socket says 172.21.15.18:42288) 2024-03-29T11:10:28.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.039+0000 7f338359e640 10 monclient(hunting): picked mon.noname-a con 0x7f337c0c5120 addr v1:172.21.15.18:6789/0 2024-03-29T11:10:28.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.039+0000 7f338359e640 10 monclient(hunting): picked mon.noname-c con 0x7f337c0cb780 addr v1:172.21.15.70:6789/0 2024-03-29T11:10:28.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.039+0000 7f338359e640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f337c008ed0 con 0x7f337c0c5120 2024-03-29T11:10:28.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.039+0000 7f338359e640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f337c02ffb0 con 0x7f337c0cb780 2024-03-29T11:10:28.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.039+0000 7f338359e640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f337c0cbc10 con 0x7f337c006ad0 2024-03-29T11:10:28.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.039+0000 7f338359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:28.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.039+0000 7f338359e640 10 monclient(hunting): authenticate will time out at 854.311609s 2024-03-29T11:10:28.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.039+0000 7f338259c640 1 -- 172.21.15.18:0/3729389119 learned_addr learned my addr 172.21.15.18:0/3729389119 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:10:28.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.039+0000 7f338159a640 1 -- 172.21.15.18:0/3729389119 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 702402069 0 0) 0x7f337c0cbc10 con 0x7f337c006ad0 2024-03-29T11:10:28.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.039+0000 7f338159a640 10 monclient(hunting): _negotiate 2024-03-29T11:10:28.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.039+0000 7f338159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:28.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.039+0000 7f338159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:28.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.039+0000 7f338159a640 10 monclient(hunting): my global_id is 4286 2024-03-29T11:10:28.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.040+0000 7f338159a640 1 -- 172.21.15.18:0/3729389119 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f336c003520 con 0x7f337c006ad0 2024-03-29T11:10:28.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.040+0000 7f338159a640 1 -- 172.21.15.18:0/3729389119 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2263965873 0 0) 0x7f336c003520 con 0x7f337c006ad0 2024-03-29T11:10:28.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.040+0000 7f338159a640 1 -- 172.21.15.18:0/3729389119 >> v1:172.21.15.70:6789/0 conn(0x7f337c0cb780 legacy=0x7f337c0d3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:28.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.040+0000 7f338159a640 1 -- 172.21.15.18:0/3729389119 >> v1:172.21.15.18:6789/0 conn(0x7f337c0c5120 legacy=0x7f337c0cb070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:10:28.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.041+0000 7f338159a640 10 monclient: _finish_hunting 0 2024-03-29T11:10:28.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.041+0000 7f338159a640 1 monclient: found mon.noname-b 2024-03-29T11:10:28.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.041+0000 7f338159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:28.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.041+0000 7f338159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:10:28.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.041+0000 7f338159a640 1 -- 172.21.15.18:0/3729389119 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f337c030220 con 0x7f337c006ad0 2024-03-29T11:10:28.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.041+0000 7f338159a640 10 monclient: _finish_auth 0 2024-03-29T11:10:28.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.041+0000 7f338159a640 10 monclient: _check_auth_tickets 2024-03-29T11:10:28.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.041+0000 7f338159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:28.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.041+0000 7f338159a640 1 -- 172.21.15.18:0/3729389119 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f3374001690 con 0x7f337c006ad0 2024-03-29T11:10:28.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.041+0000 7f338159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:28.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.041+0000 7f338159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:10:28.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.041+0000 7f338159a640 10 monclient: dump: 2024-03-29T11:10:28.023 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:10:28.023 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:28.023 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:28.023 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:28.024 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:28.024 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:28.024 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:10:28.024 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:10:28.024 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:10:28.024 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:28.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.041+0000 7f338359e640 5 monclient: authenticate success, global_id 4286 2024-03-29T11:10:28.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.041+0000 7f338359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:10:28.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.041+0000 7f338159a640 1 -- 172.21.15.18:0/3729389119 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3374001860 con 0x7f337c006ad0 2024-03-29T11:10:28.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.041+0000 7f338159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:28.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.041+0000 7f338159a640 1 -- 172.21.15.18:0/3729389119 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f3374001c70 con 0x7f337c006ad0 2024-03-29T11:10:28.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.041+0000 7f338159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:28.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.041+0000 7f338159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:10:28.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.041+0000 7f338159a640 10 monclient: dump: 2024-03-29T11:10:28.024 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:10:28.025 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:28.025 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:28.025 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:28.025 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:28.025 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:28.025 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:10:28.025 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:10:28.025 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:10:28.025 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:28.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.042+0000 7f338359e640 10 monclient: get_monmap_and_config success 2024-03-29T11:10:28.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.042+0000 7f338359e640 4 set_mon_vals no callback set 2024-03-29T11:10:28.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.043+0000 7f338359e640 10 monclient: shutdown 2024-03-29T11:10:28.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.043+0000 7f338359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:28.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.043+0000 7f338359e640 1 -- 172.21.15.18:0/3729389119 >> v1:172.21.15.18:6790/0 conn(0x7f337c006ad0 legacy=0x7f337c0c4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:28.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.043+0000 7f338359e640 1 -- 172.21.15.18:0/3729389119 shutdown_connections 2024-03-29T11:10:28.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.043+0000 7f338359e640 1 -- 172.21.15.18:0/3729389119 >> 172.21.15.18:0/3729389119 conn(0x7f337c0058f0 msgr2=0x7f337c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:28.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.043+0000 7f338359e640 1 -- 172.21.15.18:0/3729389119 shutdown_connections 2024-03-29T11:10:28.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.043+0000 7f338359e640 1 -- 172.21.15.18:0/3729389119 wait complete. 2024-03-29T11:10:28.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.044+0000 7f338359e640 10 monclient: build_initial_monmap 2024-03-29T11:10:28.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.044+0000 7f338359e640 10 monclient: monmap: 2024-03-29T11:10:28.026 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:10:28.026 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:10:28.026 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:10:28.026 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:10:28.026 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:28.026 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:28.026 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:10:28.026 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:10:28.026 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:10:28.026 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:28.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.045+0000 7f338359e640 1 librados: starting msgr at 2024-03-29T11:10:28.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.045+0000 7f338359e640 1 librados: starting objecter 2024-03-29T11:10:28.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.045+0000 7f338359e640 1 Processor -- start 2024-03-29T11:10:28.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.045+0000 7f338359e640 1 -- start start 2024-03-29T11:10:28.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.045+0000 7f338359e640 1 librados: setting wanted keys 2024-03-29T11:10:28.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.045+0000 7f338359e640 1 librados: calling monclient init 2024-03-29T11:10:28.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.045+0000 7f338359e640 10 monclient: init 2024-03-29T11:10:28.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.046+0000 7f338359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:28.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.046+0000 7f338359e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:10:28.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.046+0000 7f338359e640 10 monclient(hunting): picked mon.noname-b con 0x7f337c006ad0 addr v1:172.21.15.70:6789/0 2024-03-29T11:10:28.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.046+0000 7f338359e640 10 monclient(hunting): picked mon.noname-a con 0x7f337c0c5120 addr v1:172.21.15.18:6789/0 2024-03-29T11:10:28.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.047+0000 7f338359e640 10 monclient(hunting): picked mon.noname-c con 0x7f337c0cb780 addr v1:172.21.15.18:6790/0 2024-03-29T11:10:28.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.047+0000 7f338259c640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f337c006ad0 0x7f337c039f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.18:54964/0 (socket says 172.21.15.18:54964) 2024-03-29T11:10:28.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.047+0000 7f338259c640 1 -- 172.21.15.18:0/1734744889 learned_addr learned my addr 172.21.15.18:0/1734744889 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:10:28.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.047+0000 7f3372ffd640 10 client.?.objecter ms_handle_connect 0x7f337c006ad0 2024-03-29T11:10:28.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.047+0000 7f3372ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:28.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.047+0000 7f3372ffd640 10 client.?.objecter ms_handle_connect 0x7f337c0cb780 2024-03-29T11:10:28.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.047+0000 7f3372ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:28.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.047+0000 7f338359e640 1 -- 172.21.15.18:0/1734744889 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f337c03ab80 con 0x7f337c0c5120 2024-03-29T11:10:28.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.047+0000 7f338359e640 1 -- 172.21.15.18:0/1734744889 --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f337c039290 con 0x7f337c006ad0 2024-03-29T11:10:28.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.047+0000 7f338359e640 1 -- 172.21.15.18:0/1734744889 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f337c039500 con 0x7f337c0cb780 2024-03-29T11:10:28.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.047+0000 7f338359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:28.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.047+0000 7f338359e640 10 monclient(hunting): authenticate will time out at 854.319874s 2024-03-29T11:10:28.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.047+0000 7f3372ffd640 1 -- 172.21.15.18:0/1734744889 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4168207006 0 0) 0x7f337c039290 con 0x7f337c006ad0 2024-03-29T11:10:28.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.048+0000 7f3372ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:10:28.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.048+0000 7f3372ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:28.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.048+0000 7f3372ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:28.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.048+0000 7f3372ffd640 10 monclient(hunting): my global_id is 4282 2024-03-29T11:10:28.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.048+0000 7f3372ffd640 1 -- 172.21.15.18:0/1734744889 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3364004eb0 con 0x7f337c006ad0 2024-03-29T11:10:28.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.048+0000 7f3372ffd640 1 -- 172.21.15.18:0/1734744889 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2268995885 0 0) 0x7f337c039500 con 0x7f337c0cb780 2024-03-29T11:10:28.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.048+0000 7f3372ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:10:28.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.048+0000 7f3372ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:28.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.048+0000 7f3372ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:28.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.048+0000 7f3372ffd640 10 monclient(hunting): my global_id is 4289 2024-03-29T11:10:28.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.048+0000 7f3372ffd640 1 -- 172.21.15.18:0/1734744889 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f337c039290 con 0x7f337c0cb780 2024-03-29T11:10:28.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.048+0000 7f3372ffd640 1 -- 172.21.15.18:0/1734744889 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 87116153 0 0) 0x7f3364004eb0 con 0x7f337c006ad0 2024-03-29T11:10:28.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.048+0000 7f3372ffd640 1 -- 172.21.15.18:0/1734744889 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f337c039500 con 0x7f337c006ad0 2024-03-29T11:10:28.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.048+0000 7f3372ffd640 1 -- 172.21.15.18:0/1734744889 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f3374001b70 con 0x7f337c006ad0 2024-03-29T11:10:28.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.048+0000 7f3372ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:28.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.048+0000 7f3372ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:10:28.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.049+0000 7f3372ffd640 10 monclient(hunting): dump: 2024-03-29T11:10:28.037 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:10:28.037 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:28.037 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:28.037 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:28.038 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:28.038 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:28.038 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:10:28.038 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:10:28.038 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:10:28.038 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:28.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.049+0000 7f3372ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:10:28.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.049+0000 7f3372ffd640 1 -- 172.21.15.18:0/1734744889 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1558370024 0 0) 0x7f337c039290 con 0x7f337c0cb780 2024-03-29T11:10:28.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.049+0000 7f3372ffd640 1 -- 172.21.15.18:0/1734744889 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3364004eb0 con 0x7f337c0cb780 2024-03-29T11:10:28.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.049+0000 7f3372ffd640 1 -- 172.21.15.18:0/1734744889 <== mon.1 v1:172.21.15.70:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2072693192 0 0) 0x7f337c039500 con 0x7f337c006ad0 2024-03-29T11:10:28.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.049+0000 7f3372ffd640 1 -- 172.21.15.18:0/1734744889 >> v1:172.21.15.18:6790/0 conn(0x7f337c0cb780 legacy=0x7f337c035fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:28.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.049+0000 7f3372ffd640 1 -- 172.21.15.18:0/1734744889 >> v1:172.21.15.18:6789/0 conn(0x7f337c0c5120 legacy=0x7f337c035890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:28.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.049+0000 7f3372ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:10:28.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.049+0000 7f3372ffd640 1 monclient: found mon.b 2024-03-29T11:10:28.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.049+0000 7f3372ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:28.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.049+0000 7f3372ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:10:28.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.049+0000 7f3372ffd640 1 -- 172.21.15.18:0/1734744889 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f337c0ca740 con 0x7f337c006ad0 2024-03-29T11:10:28.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.049+0000 7f3372ffd640 10 monclient: _finish_auth 0 2024-03-29T11:10:28.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.049+0000 7f3372ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:10:28.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.049+0000 7f3372ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:28.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.049+0000 7f3372ffd640 10 client.?.objecter ms_handle_connect 0x7f337c0c5120 2024-03-29T11:10:28.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.049+0000 7f3372ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:28.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.050+0000 7f338359e640 5 monclient: authenticate success, global_id 4282 2024-03-29T11:10:28.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.050+0000 7f338359e640 10 monclient: _renew_subs 2024-03-29T11:10:28.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.050+0000 7f338359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:10:28.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.050+0000 7f338359e640 1 -- 172.21.15.18:0/1734744889 --> v1:172.21.15.70:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f337c03a680 con 0x7f337c006ad0 2024-03-29T11:10:28.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.050+0000 7f338359e640 10 client.4282.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:10:28.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.050+0000 7f338359e640 10 monclient: _renew_subs 2024-03-29T11:10:28.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.050+0000 7f338359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:10:28.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.050+0000 7f338359e640 1 -- 172.21.15.18:0/1734744889 --> v1:172.21.15.70:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f337c168980 con 0x7f337c006ad0 2024-03-29T11:10:28.040 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.050+0000 7f338359e640 1 librados: init done 2024-03-29T11:10:28.040 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.051+0000 7f3372ffd640 1 -- 172.21.15.18:0/1734744889 <== mon.1 v1:172.21.15.70:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3374004bf0 con 0x7f337c006ad0 2024-03-29T11:10:28.040 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.051+0000 7f3372ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:28.040 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.051+0000 7f3372ffd640 1 -- 172.21.15.18:0/1734744889 <== mon.1 v1:172.21.15.70:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f3374004f00 con 0x7f337c006ad0 2024-03-29T11:10:28.040 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.051+0000 7f3372ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:28.040 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.051+0000 7f3372ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:10:28.040 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.051+0000 7f3372ffd640 10 monclient: dump: 2024-03-29T11:10:28.040 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:10:28.040 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:28.040 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:28.040 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:28.040 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:28.040 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:28.040 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:10:28.041 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:10:28.041 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:10:28.041 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:28.041 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.051+0000 7f338359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:28.041 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.051+0000 7f338359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:28.041 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.051+0000 7f338359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:10:28.041 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.051+0000 7f338359e640 1 -- 172.21.15.18:0/1734744889 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f337c0bf1c0 con 0x7f337c006ad0 2024-03-29T11:10:28.041 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.052+0000 7f3380d99640 4 set_mon_vals no callback set 2024-03-29T11:10:28.041 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.052+0000 7f3372ffd640 1 -- 172.21.15.18:0/1734744889 <== mon.1 v1:172.21.15.70:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f337401aed0 con 0x7f337c006ad0 2024-03-29T11:10:28.041 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.053+0000 7f3372ffd640 1 -- 172.21.15.18:0/1734744889 <== mon.1 v1:172.21.15.70:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f3374084bf0 con 0x7f337c006ad0 2024-03-29T11:10:28.041 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.053+0000 7f3372ffd640 10 client.4282.objecter ms_dispatch 0x7f3384000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:10:28.041 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.053+0000 7f3372ffd640 3 client.4282.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:10:28.041 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.053+0000 7f3372ffd640 3 client.4282.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:10:28.041 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.053+0000 7f3372ffd640 20 client.4282.objecter dump_active .. 0 homeless 2024-03-29T11:10:28.041 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.053+0000 7f3381d9b640 10 monclient: get_auth_request con 0x7f336406dca0 auth_method 0 2024-03-29T11:10:28.041 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.053+0000 7f3372ffd640 10 client.4282.objecter ms_handle_connect 0x7f336406dca0 2024-03-29T11:10:28.042 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.060+0000 7f3372ffd640 1 -- 172.21.15.18:0/1734744889 <== mon.1 v1:172.21.15.70:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f33740504d0 con 0x7f337c006ad0 2024-03-29T11:10:28.042 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.060+0000 7f3372ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:28.042 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.060+0000 7f3372ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:10:28.355 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.376+0000 7f338359e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:28.355 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.377+0000 7f338359e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:28.355 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.377+0000 7f338359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:10:28.355 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.377+0000 7f338359e640 1 -- 172.21.15.18:0/1734744889 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f337c004cd0 con 0x7f337c006ad0 2024-03-29T11:10:28.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.378+0000 7f3372ffd640 1 -- 172.21.15.18:0/1734744889 <== mon.1 v1:172.21.15.70:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f3374052400 con 0x7f337c006ad0 2024-03-29T11:10:28.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.378+0000 7f3372ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:28.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.378+0000 7f3372ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:10:28.356 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:10:28.356 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:10:28.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.381+0000 7f338359e640 10 librados: watch_flush enter 2024-03-29T11:10:28.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.381+0000 7f338359e640 10 librados: watch_flush exit 2024-03-29T11:10:28.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.381+0000 7f338359e640 20 client.4282.objecter shutdown clearing up homeless session... 2024-03-29T11:10:28.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.381+0000 7f338359e640 10 client.4282.objecter successfully canceled tick 2024-03-29T11:10:28.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.381+0000 7f338359e640 1 -- 172.21.15.18:0/1734744889 >> v1:172.21.15.70:6816/3101871562 conn(0x7f336406dca0 legacy=0x7f3364070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:28.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.381+0000 7f338359e640 10 monclient: shutdown 2024-03-29T11:10:28.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.381+0000 7f338359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:28.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.381+0000 7f338359e640 1 -- 172.21.15.18:0/1734744889 >> v1:172.21.15.70:6789/0 conn(0x7f337c006ad0 legacy=0x7f337c039f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:28.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.381+0000 7f338359e640 1 -- 172.21.15.18:0/1734744889 shutdown_connections 2024-03-29T11:10:28.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.381+0000 7f338359e640 1 -- 172.21.15.18:0/1734744889 >> 172.21.15.18:0/1734744889 conn(0x7f337c0058f0 msgr2=0x7f337c0043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:28.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.381+0000 7f338359e640 1 -- 172.21.15.18:0/1734744889 shutdown_connections 2024-03-29T11:10:28.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.381+0000 7f338359e640 1 -- 172.21.15.18:0/1734744889 wait complete. 2024-03-29T11:10:28.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:28.381+0000 7f338359e640 1 librados: shutdown 2024-03-29T11:10:28.373 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:10:35.181 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:10:35.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.351+0000 7f494c9f5640 10 monclient: get_monmap_and_config 2024-03-29T11:10:35.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.351+0000 7f494c9f5640 10 monclient: build_initial_monmap 2024-03-29T11:10:35.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.351+0000 7f494c9f5640 10 monclient: monmap: 2024-03-29T11:10:35.330 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:10:35.330 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:35.330 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:10:35.352540+0000 2024-03-29T11:10:35.330 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:10:35.352540+0000 2024-03-29T11:10:35.330 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:35.330 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:35.330 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:10:35.330 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:10:35.331 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:10:35.331 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:35.331 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.352+0000 7f494c9f5640 1 Processor -- start 2024-03-29T11:10:35.331 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.352+0000 7f494c9f5640 1 -- start start 2024-03-29T11:10:35.332 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.352+0000 7f494c9f5640 10 monclient: init 2024-03-29T11:10:35.332 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.353+0000 7f494c9f5640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:35.332 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.353+0000 7f494c9f5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:10:35.332 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.353+0000 7f494c9f5640 10 monclient(hunting): picked mon.noname-a con 0x7f4940006e30 addr v1:172.21.15.18:6789/0 2024-03-29T11:10:35.332 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.353+0000 7f494c9f5640 10 monclient(hunting): picked mon.noname-c con 0x7f4940035e00 addr v1:172.21.15.70:6789/0 2024-03-29T11:10:35.332 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.353+0000 7f494c9f5640 10 monclient(hunting): picked mon.noname-b con 0x7f49400c9670 addr v1:172.21.15.18:6790/0 2024-03-29T11:10:35.332 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.353+0000 7f494c9f5640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4940035970 con 0x7f4940006e30 2024-03-29T11:10:35.332 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.353+0000 7f494c9f5640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4940036280 con 0x7f4940035e00 2024-03-29T11:10:35.332 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.353+0000 7f494c9f5640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4940035640 con 0x7f49400c9670 2024-03-29T11:10:35.332 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.353+0000 7f494c9f5640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:35.332 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.353+0000 7f494c9f5640 10 monclient(hunting): authenticate will time out at 861.625676s 2024-03-29T11:10:35.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.356+0000 7f494759e640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f49400c9670 0x7f49400d7560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:49387/0 (socket says 172.21.15.18:49387) 2024-03-29T11:10:35.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.356+0000 7f494759e640 1 -- 172.21.15.18:0/414614917 learned_addr learned my addr 172.21.15.18:0/414614917 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:10:35.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.357+0000 7f4945d9b640 1 -- 172.21.15.18:0/414614917 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4153556228 0 0) 0x7f4940035640 con 0x7f49400c9670 2024-03-29T11:10:35.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.357+0000 7f4945d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:10:35.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.357+0000 7f4945d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:35.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.357+0000 7f4945d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:35.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.357+0000 7f4945d9b640 10 monclient(hunting): my global_id is 4292 2024-03-29T11:10:35.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.357+0000 7f4945d9b640 1 -- 172.21.15.18:0/414614917 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4930003520 con 0x7f49400c9670 2024-03-29T11:10:35.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.357+0000 7f4945d9b640 1 -- 172.21.15.18:0/414614917 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1884022512 0 0) 0x7f4940036280 con 0x7f4940035e00 2024-03-29T11:10:35.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.357+0000 7f4945d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:10:35.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.357+0000 7f4945d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:35.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.357+0000 7f4945d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:35.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.357+0000 7f4945d9b640 10 monclient(hunting): my global_id is 4285 2024-03-29T11:10:35.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.357+0000 7f4945d9b640 1 -- 172.21.15.18:0/414614917 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4940035640 con 0x7f4940035e00 2024-03-29T11:10:35.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.358+0000 7f4945d9b640 1 -- 172.21.15.18:0/414614917 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3826520337 0 0) 0x7f4930003520 con 0x7f49400c9670 2024-03-29T11:10:35.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.358+0000 7f4945d9b640 1 -- 172.21.15.18:0/414614917 >> v1:172.21.15.70:6789/0 conn(0x7f4940035e00 legacy=0x7f49400c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:35.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.358+0000 7f4945d9b640 1 -- 172.21.15.18:0/414614917 >> v1:172.21.15.18:6789/0 conn(0x7f4940006e30 legacy=0x7f49400cbcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:10:35.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.358+0000 7f4945d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:10:35.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.358+0000 7f4945d9b640 1 monclient: found mon.noname-b 2024-03-29T11:10:35.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.358+0000 7f4945d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:35.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.358+0000 7f4945d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:10:35.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.358+0000 7f4945d9b640 1 -- 172.21.15.18:0/414614917 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4940034e20 con 0x7f49400c9670 2024-03-29T11:10:35.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.358+0000 7f4945d9b640 10 monclient: _finish_auth 0 2024-03-29T11:10:35.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.358+0000 7f4945d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:10:35.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.358+0000 7f4945d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:35.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.358+0000 7f494c9f5640 5 monclient: authenticate success, global_id 4292 2024-03-29T11:10:35.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.358+0000 7f494c9f5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:10:35.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.358+0000 7f4945d9b640 1 -- 172.21.15.18:0/414614917 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f4938001690 con 0x7f49400c9670 2024-03-29T11:10:35.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.358+0000 7f4945d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:35.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.358+0000 7f4945d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:10:35.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.358+0000 7f4945d9b640 10 monclient: dump: 2024-03-29T11:10:35.340 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:10:35.340 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:35.341 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:35.341 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:35.341 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:35.341 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:35.341 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:10:35.341 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:10:35.341 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:10:35.341 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:35.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.358+0000 7f494c9f5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:10:35.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.359+0000 7f4945d9b640 1 -- 172.21.15.18:0/414614917 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4938001920 con 0x7f49400c9670 2024-03-29T11:10:35.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.359+0000 7f4945d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:35.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.359+0000 7f494c9f5640 10 monclient: get_monmap_and_config success 2024-03-29T11:10:35.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.359+0000 7f494c9f5640 4 set_mon_vals no callback set 2024-03-29T11:10:35.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.360+0000 7f4945d9b640 1 -- 172.21.15.18:0/414614917 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f4938001cb0 con 0x7f49400c9670 2024-03-29T11:10:35.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.360+0000 7f4945d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:35.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.360+0000 7f4945d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:10:35.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.360+0000 7f4945d9b640 10 monclient: dump: 2024-03-29T11:10:35.342 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:10:35.342 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:35.342 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:35.342 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:35.342 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:35.342 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:35.342 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:10:35.342 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:10:35.342 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:10:35.342 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:35.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.360+0000 7f494c9f5640 10 monclient: shutdown 2024-03-29T11:10:35.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.360+0000 7f494c9f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:35.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.360+0000 7f494c9f5640 1 -- 172.21.15.18:0/414614917 >> v1:172.21.15.18:6790/0 conn(0x7f49400c9670 legacy=0x7f49400d7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:35.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.361+0000 7f494c9f5640 1 -- 172.21.15.18:0/414614917 shutdown_connections 2024-03-29T11:10:35.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.361+0000 7f494c9f5640 1 -- 172.21.15.18:0/414614917 >> 172.21.15.18:0/414614917 conn(0x7f4940005c00 msgr2=0x7f4940006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:35.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.361+0000 7f494c9f5640 1 -- 172.21.15.18:0/414614917 shutdown_connections 2024-03-29T11:10:35.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.361+0000 7f494c9f5640 1 -- 172.21.15.18:0/414614917 wait complete. 2024-03-29T11:10:35.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.362+0000 7f494c9f5640 10 monclient: build_initial_monmap 2024-03-29T11:10:35.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.362+0000 7f494c9f5640 10 monclient: monmap: 2024-03-29T11:10:35.345 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:10:35.345 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:10:35.345 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:10:35.345 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:10:35.345 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:35.345 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:35.345 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:10:35.345 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:10:35.346 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:10:35.346 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:35.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.362+0000 7f494c9f5640 1 librados: starting msgr at 2024-03-29T11:10:35.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.363+0000 7f494c9f5640 1 librados: starting objecter 2024-03-29T11:10:35.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.363+0000 7f494c9f5640 1 Processor -- start 2024-03-29T11:10:35.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.363+0000 7f494c9f5640 1 -- start start 2024-03-29T11:10:35.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.363+0000 7f494c9f5640 1 librados: setting wanted keys 2024-03-29T11:10:35.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.363+0000 7f494c9f5640 1 librados: calling monclient init 2024-03-29T11:10:35.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.363+0000 7f494c9f5640 10 monclient: init 2024-03-29T11:10:35.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.364+0000 7f494c9f5640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:35.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.364+0000 7f494c9f5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:10:35.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.364+0000 7f494c9f5640 10 monclient(hunting): picked mon.noname-c con 0x7f4940006e30 addr v1:172.21.15.18:6790/0 2024-03-29T11:10:35.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.364+0000 7f494c9f5640 10 monclient(hunting): picked mon.noname-a con 0x7f4940035e00 addr v1:172.21.15.18:6789/0 2024-03-29T11:10:35.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.364+0000 7f494c9f5640 10 monclient(hunting): picked mon.noname-b con 0x7f49400c9670 addr v1:172.21.15.70:6789/0 2024-03-29T11:10:35.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.365+0000 7f494c9f5640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f494002f2c0 con 0x7f4940035e00 2024-03-29T11:10:35.347 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.365+0000 7f494c9f5640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49400c4fa0 con 0x7f49400c9670 2024-03-29T11:10:35.347 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.365+0000 7f494c9f5640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49400cfdd0 con 0x7f4940006e30 2024-03-29T11:10:35.347 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.365+0000 7f494c9f5640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:35.347 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.365+0000 7f494c9f5640 10 monclient(hunting): authenticate will time out at 861.637366s 2024-03-29T11:10:35.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.366+0000 7f494759e640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f49400c9670 0x7f494016bad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.18:52708/0 (socket says 172.21.15.18:52708) 2024-03-29T11:10:35.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.366+0000 7f494759e640 1 -- 172.21.15.18:0/935103106 learned_addr learned my addr 172.21.15.18:0/935103106 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:10:35.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.366+0000 7f492f7fe640 10 client.?.objecter ms_handle_connect 0x7f49400c9670 2024-03-29T11:10:35.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.366+0000 7f492f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:35.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.369+0000 7f492f7fe640 1 -- 172.21.15.18:0/935103106 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2895860361 0 0) 0x7f49400c4fa0 con 0x7f49400c9670 2024-03-29T11:10:35.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.369+0000 7f492f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:10:35.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.369+0000 7f492f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:35.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.369+0000 7f492f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:35.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.369+0000 7f492f7fe640 10 monclient(hunting): my global_id is 4288 2024-03-29T11:10:35.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.369+0000 7f492f7fe640 1 -- 172.21.15.18:0/935103106 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4934004eb0 con 0x7f49400c9670 2024-03-29T11:10:35.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.369+0000 7f492f7fe640 10 client.?.objecter ms_handle_connect 0x7f4940006e30 2024-03-29T11:10:35.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.369+0000 7f492f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:35.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.369+0000 7f492f7fe640 1 -- 172.21.15.18:0/935103106 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 87026418 0 0) 0x7f49400cfdd0 con 0x7f4940006e30 2024-03-29T11:10:35.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.369+0000 7f492f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:10:35.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.369+0000 7f492f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:35.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.370+0000 7f492f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:35.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.370+0000 7f492f7fe640 10 monclient(hunting): my global_id is 4295 2024-03-29T11:10:35.355 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.370+0000 7f492f7fe640 1 -- 172.21.15.18:0/935103106 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49400c4fa0 con 0x7f4940006e30 2024-03-29T11:10:35.355 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.370+0000 7f492f7fe640 1 -- 172.21.15.18:0/935103106 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1344617461 0 0) 0x7f4934004eb0 con 0x7f49400c9670 2024-03-29T11:10:35.355 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.370+0000 7f492f7fe640 1 -- 172.21.15.18:0/935103106 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49400cfdd0 con 0x7f49400c9670 2024-03-29T11:10:35.355 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.370+0000 7f492f7fe640 1 -- 172.21.15.18:0/935103106 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f4938001550 con 0x7f49400c9670 2024-03-29T11:10:35.355 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.370+0000 7f492f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:35.355 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.370+0000 7f492f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:10:35.355 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.370+0000 7f492f7fe640 10 monclient(hunting): dump: 2024-03-29T11:10:35.355 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:10:35.355 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:35.355 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:35.355 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:35.355 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:35.355 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:35.355 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:10:35.356 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:10:35.356 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:10:35.356 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:35.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.371+0000 7f492f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:10:35.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.371+0000 7f492f7fe640 1 -- 172.21.15.18:0/935103106 <== mon.1 v1:172.21.15.70:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1142064186 0 0) 0x7f49400cfdd0 con 0x7f49400c9670 2024-03-29T11:10:35.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.371+0000 7f492f7fe640 1 -- 172.21.15.18:0/935103106 >> v1:172.21.15.18:6790/0 conn(0x7f4940006e30 legacy=0x7f49400cda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:35.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.371+0000 7f492f7fe640 1 -- 172.21.15.18:0/935103106 >> v1:172.21.15.18:6789/0 conn(0x7f4940035e00 legacy=0x7f49400ce4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:35.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.371+0000 7f492f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:10:35.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.371+0000 7f492f7fe640 1 monclient: found mon.b 2024-03-29T11:10:35.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.371+0000 7f492f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:35.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.371+0000 7f492f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:10:35.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.371+0000 7f492f7fe640 1 -- 172.21.15.18:0/935103106 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49400c53c0 con 0x7f49400c9670 2024-03-29T11:10:35.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.371+0000 7f492f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:10:35.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.371+0000 7f492f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:10:35.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.372+0000 7f492f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:35.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.372+0000 7f492f7fe640 1 -- 172.21.15.18:0/935103106 <== mon.1 v1:172.21.15.70:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49380016b0 con 0x7f49400c9670 2024-03-29T11:10:35.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.372+0000 7f494c9f5640 5 monclient: authenticate success, global_id 4288 2024-03-29T11:10:35.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.372+0000 7f494c9f5640 10 monclient: _renew_subs 2024-03-29T11:10:35.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.372+0000 7f494c9f5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:10:35.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.372+0000 7f494c9f5640 1 -- 172.21.15.18:0/935103106 --> v1:172.21.15.70:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f494002ff70 con 0x7f49400c9670 2024-03-29T11:10:35.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.372+0000 7f494c9f5640 10 client.4288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:10:35.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.372+0000 7f494c9f5640 10 monclient: _renew_subs 2024-03-29T11:10:35.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.372+0000 7f494c9f5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:10:35.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.372+0000 7f494c9f5640 1 -- 172.21.15.18:0/935103106 --> v1:172.21.15.70:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4940033690 con 0x7f49400c9670 2024-03-29T11:10:35.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.372+0000 7f494c9f5640 1 librados: init done 2024-03-29T11:10:35.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.372+0000 7f492f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:35.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.372+0000 7f492f7fe640 1 -- 172.21.15.18:0/935103106 <== mon.1 v1:172.21.15.70:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f4938004d90 con 0x7f49400c9670 2024-03-29T11:10:35.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.372+0000 7f492f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:35.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.372+0000 7f492f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:10:35.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.372+0000 7f492f7fe640 10 monclient: dump: 2024-03-29T11:10:35.358 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:10:35.358 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:35.358 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:35.358 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:35.358 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:35.358 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:35.358 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:10:35.358 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:10:35.358 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:10:35.359 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:35.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.373+0000 7f494559a640 4 set_mon_vals no callback set 2024-03-29T11:10:35.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.373+0000 7f494759e640 1 -- 172.21.15.18:0/935103106 >> v1:172.21.15.70:6789/0 conn(0x7f49400c9670 legacy=0x7f494016bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-29T11:10:35.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.373+0000 7f494759e640 1 -- 172.21.15.18:0/935103106 >> v1:172.21.15.70:6789/0 conn(0x7f49400c9670 legacy=0x7f494016bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:10:35.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.373+0000 7f494759e640 1 --1- 172.21.15.18:0/935103106 >> v1:172.21.15.70:6789/0 conn(0x7f49400c9670 0x7f494016bad0 :-1 s=OPENED pgs=203340 cs=1 l=1).handle_message read tag failed 2024-03-29T11:10:35.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.373+0000 7f494759e640 1 --1- 172.21.15.18:0/935103106 >> v1:172.21.15.70:6789/0 conn(0x7f49400c9670 0x7f494016bad0 :-1 s=OPENED pgs=203340 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:10:35.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.373+0000 7f492f7fe640 10 monclient: ms_handle_reset current mon v1:172.21.15.70:6789/0 2024-03-29T11:10:35.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.373+0000 7f492f7fe640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:35.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.373+0000 7f492f7fe640 1 -- 172.21.15.18:0/935103106 >> v1:172.21.15.70:6789/0 conn(0x7f49400c9670 legacy=0x7f494016bad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:10:35.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.374+0000 7f494c9f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:35.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.374+0000 7f492f7fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:10:35.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.374+0000 7f492f7fe640 10 monclient(hunting): picked mon.a con 0x7f4940035e00 addr v1:172.21.15.18:6789/0 2024-03-29T11:10:35.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.374+0000 7f492f7fe640 10 monclient(hunting): picked mon.b con 0x7f4934006b70 addr v1:172.21.15.70:6789/0 2024-03-29T11:10:35.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.374+0000 7f492f7fe640 10 monclient(hunting): picked mon.c con 0x7f4940006e30 addr v1:172.21.15.18:6790/0 2024-03-29T11:10:35.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.374+0000 7f492f7fe640 1 -- 172.21.15.18:0/935103106 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f49400cfdd0 con 0x7f4940035e00 2024-03-29T11:10:35.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.374+0000 7f492f7fe640 1 -- 172.21.15.18:0/935103106 --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f4934004eb0 con 0x7f4934006b70 2024-03-29T11:10:35.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.374+0000 7f492f7fe640 1 -- 172.21.15.18:0/935103106 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f49340139f0 con 0x7f4940006e30 2024-03-29T11:10:35.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.374+0000 7f492f7fe640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:35.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.374+0000 7f494c9f5640 10 monclient(hunting): _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:35.364 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.378+0000 7f492f7fe640 10 client.4288.objecter ms_handle_connect 0x7f4940006e30 2024-03-29T11:10:35.364 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.378+0000 7f492f7fe640 10 client.4288.objecter resend_mon_ops 2024-03-29T11:10:35.364 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.378+0000 7f492f7fe640 1 -- 172.21.15.18:0/935103106 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 764795310 0 0) 0x7f49340139f0 con 0x7f4940006e30 2024-03-29T11:10:35.364 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.378+0000 7f492f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:10:35.364 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.378+0000 7f492f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:35.364 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.378+0000 7f492f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T11:10:35.364 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.379+0000 7f492f7fe640 1 -- 172.21.15.18:0/935103106 --> v1:172.21.15.18:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f4934013b80 con 0x7f4940006e30 2024-03-29T11:10:35.364 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.379+0000 7f492f7fe640 1 -- 172.21.15.18:0/935103106 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 2241879208 0 0) 0x7f4934013b80 con 0x7f4940006e30 2024-03-29T11:10:35.364 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.379+0000 7f492f7fe640 1 -- 172.21.15.18:0/935103106 >> v1:172.21.15.70:6789/0 conn(0x7f4934006b70 legacy=0x7f493400c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:35.364 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.379+0000 7f492f7fe640 1 -- 172.21.15.18:0/935103106 >> v1:172.21.15.18:6789/0 conn(0x7f4940035e00 legacy=0x7f49340061f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:35.365 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.379+0000 7f492f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:10:35.365 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.379+0000 7f492f7fe640 1 monclient: found mon.c 2024-03-29T11:10:35.365 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.379+0000 7f492f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:35.365 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.379+0000 7f492f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:10:35.365 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.379+0000 7f492f7fe640 1 -- 172.21.15.18:0/935103106 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f4934014dd0 con 0x7f4940006e30 2024-03-29T11:10:35.365 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.379+0000 7f492f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:10:35.365 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.379+0000 7f492f7fe640 1 -- 172.21.15.18:0/935103106 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4928005080 con 0x7f4940006e30 2024-03-29T11:10:35.365 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.379+0000 7f492f7fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:35.365 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.379+0000 7f492f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:10:35.365 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.379+0000 7f492f7fe640 1 -- 172.21.15.18:0/935103106 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49400cfdd0 con 0x7f4940006e30 2024-03-29T11:10:35.365 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.379+0000 7f492f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:10:35.365 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.379+0000 7f492f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:10:35.365 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.379+0000 7f492f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:35.365 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.379+0000 7f492f7fe640 1 -- 172.21.15.18:0/935103106 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f4938005320 con 0x7f4940006e30 2024-03-29T11:10:35.366 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.379+0000 7f492f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:35.366 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.379+0000 7f492f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:10:35.366 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.379+0000 7f492f7fe640 10 monclient: dump: 2024-03-29T11:10:35.366 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:10:35.366 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:35.366 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:35.366 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:35.366 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:35.366 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:35.366 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:10:35.366 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:10:35.366 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:10:35.366 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:35.366 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.380+0000 7f492f7fe640 1 -- 172.21.15.18:0/935103106 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4938005480 con 0x7f4940006e30 2024-03-29T11:10:35.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.380+0000 7f492f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:35.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.380+0000 7f4944d99640 4 set_mon_vals no callback set 2024-03-29T11:10:35.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.382+0000 7f492f7fe640 1 -- 172.21.15.18:0/935103106 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f493801b520 con 0x7f4940006e30 2024-03-29T11:10:35.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.384+0000 7f492f7fe640 1 -- 172.21.15.18:0/935103106 <== mon.2 v1:172.21.15.18:6790/0 6 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f4938085350 con 0x7f4940006e30 2024-03-29T11:10:35.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.384+0000 7f4946d9d640 10 monclient: get_auth_request con 0x7f49400c9670 auth_method 0 2024-03-29T11:10:35.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.384+0000 7f492f7fe640 10 client.4288.objecter ms_dispatch 0x7f4948000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:10:35.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.384+0000 7f492f7fe640 3 client.4288.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:10:35.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.384+0000 7f492f7fe640 3 client.4288.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:10:35.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.384+0000 7f492f7fe640 20 client.4288.objecter dump_active .. 0 homeless 2024-03-29T11:10:35.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.387+0000 7f492f7fe640 10 client.4288.objecter ms_handle_connect 0x7f49400c9670 2024-03-29T11:10:35.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.389+0000 7f492f7fe640 1 -- 172.21.15.18:0/935103106 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4938050a60 con 0x7f4940006e30 2024-03-29T11:10:35.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.389+0000 7f492f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:35.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.389+0000 7f492f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:10:35.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.396+0000 7f492f7fe640 1 -- 172.21.15.18:0/935103106 <== mon.2 v1:172.21.15.18:6790/0 8 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4938052b60 con 0x7f4940006e30 2024-03-29T11:10:35.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.396+0000 7f492f7fe640 10 monclient: handle_mon_command_ack 1 not found 2024-03-29T11:10:35.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.717+0000 7f494c9f5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:35.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.717+0000 7f494c9f5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:35.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.717+0000 7f494c9f5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:10:35.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.717+0000 7f494c9f5640 1 -- 172.21.15.18:0/935103106 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49280040d0 con 0x7f4940006e30 2024-03-29T11:10:35.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.718+0000 7f492f7fe640 1 -- 172.21.15.18:0/935103106 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f4938054380 con 0x7f4940006e30 2024-03-29T11:10:35.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.718+0000 7f492f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:35.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.718+0000 7f492f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:10:35.697 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:10:35.697 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:10:35.701 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.722+0000 7f494c9f5640 10 librados: watch_flush enter 2024-03-29T11:10:35.701 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.722+0000 7f494c9f5640 10 librados: watch_flush exit 2024-03-29T11:10:35.701 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.722+0000 7f494c9f5640 20 client.4288.objecter shutdown clearing up homeless session... 2024-03-29T11:10:35.701 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.722+0000 7f494c9f5640 10 client.4288.objecter successfully canceled tick 2024-03-29T11:10:35.701 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.722+0000 7f494c9f5640 1 -- 172.21.15.18:0/935103106 >> v1:172.21.15.70:6816/3101871562 conn(0x7f49400c9670 legacy=0x7f493407b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:35.701 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.722+0000 7f494c9f5640 10 monclient: shutdown 2024-03-29T11:10:35.701 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.722+0000 7f494c9f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:35.701 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.722+0000 7f494c9f5640 1 -- 172.21.15.18:0/935103106 >> v1:172.21.15.18:6790/0 conn(0x7f4940006e30 legacy=0x7f493400fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:35.701 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.722+0000 7f494c9f5640 1 -- 172.21.15.18:0/935103106 shutdown_connections 2024-03-29T11:10:35.701 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.722+0000 7f494c9f5640 1 -- 172.21.15.18:0/935103106 >> 172.21.15.18:0/935103106 conn(0x7f4940005c00 msgr2=0x7f49400c8d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:35.701 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.722+0000 7f494c9f5640 1 -- 172.21.15.18:0/935103106 shutdown_connections 2024-03-29T11:10:35.701 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.723+0000 7f494c9f5640 1 -- 172.21.15.18:0/935103106 wait complete. 2024-03-29T11:10:35.701 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:35.723+0000 7f494c9f5640 1 librados: shutdown 2024-03-29T11:10:35.718 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:10:42.525 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:10:42.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.692+0000 7fbd1db04640 10 monclient: get_monmap_and_config 2024-03-29T11:10:42.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.692+0000 7fbd1db04640 10 monclient: build_initial_monmap 2024-03-29T11:10:42.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.693+0000 7fbd1db04640 10 monclient: monmap: 2024-03-29T11:10:42.671 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:10:42.671 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:42.671 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:10:42.693917+0000 2024-03-29T11:10:42.672 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:10:42.693917+0000 2024-03-29T11:10:42.672 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:42.672 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:42.672 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:10:42.672 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:10:42.672 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:10:42.672 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:42.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.694+0000 7fbd1db04640 1 Processor -- start 2024-03-29T11:10:42.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.694+0000 7fbd1db04640 1 -- start start 2024-03-29T11:10:42.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.694+0000 7fbd1db04640 10 monclient: init 2024-03-29T11:10:42.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.695+0000 7fbd1db04640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:42.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.695+0000 7fbd1db04640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:10:42.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.695+0000 7fbd1db04640 10 monclient(hunting): picked mon.noname-c con 0x7fbd1806d9d0 addr v1:172.21.15.70:6789/0 2024-03-29T11:10:42.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.695+0000 7fbd1db04640 10 monclient(hunting): picked mon.noname-a con 0x7fbd1813ef30 addr v1:172.21.15.18:6789/0 2024-03-29T11:10:42.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.695+0000 7fbd1db04640 10 monclient(hunting): picked mon.noname-b con 0x7fbd180b07d0 addr v1:172.21.15.18:6790/0 2024-03-29T11:10:42.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.695+0000 7fbd1db04640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd180aed10 con 0x7fbd1813ef30 2024-03-29T11:10:42.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.695+0000 7fbd1db04640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd180b0c40 con 0x7fbd1806d9d0 2024-03-29T11:10:42.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.695+0000 7fbd1db04640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd180ab1b0 con 0x7fbd180b07d0 2024-03-29T11:10:42.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.695+0000 7fbd1db04640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:42.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.695+0000 7fbd1db04640 10 monclient(hunting): authenticate will time out at 868.967366s 2024-03-29T11:10:42.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.695+0000 7fbd177fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fbd1806d9d0 0x7fbd1813e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.18:49350/0 (socket says 172.21.15.18:49350) 2024-03-29T11:10:42.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.695+0000 7fbd177fe640 1 -- 172.21.15.18:0/62260908 learned_addr learned my addr 172.21.15.18:0/62260908 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:10:42.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.696+0000 7fbd167fc640 1 -- 172.21.15.18:0/62260908 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3766730758 0 0) 0x7fbd180b0c40 con 0x7fbd1806d9d0 2024-03-29T11:10:42.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.696+0000 7fbd167fc640 10 monclient(hunting): _negotiate 2024-03-29T11:10:42.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.696+0000 7fbd167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:42.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.696+0000 7fbd167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:42.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.696+0000 7fbd167fc640 10 monclient(hunting): my global_id is 4291 2024-03-29T11:10:42.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.696+0000 7fbd167fc640 1 -- 172.21.15.18:0/62260908 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd00003520 con 0x7fbd1806d9d0 2024-03-29T11:10:42.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.696+0000 7fbd167fc640 1 -- 172.21.15.18:0/62260908 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3593739223 0 0) 0x7fbd00003520 con 0x7fbd1806d9d0 2024-03-29T11:10:42.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.696+0000 7fbd167fc640 1 -- 172.21.15.18:0/62260908 >> v1:172.21.15.18:6790/0 conn(0x7fbd180b07d0 legacy=0x7fbd1814bd80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:10:42.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.698+0000 7fbd167fc640 1 -- 172.21.15.18:0/62260908 >> v1:172.21.15.18:6789/0 conn(0x7fbd1813ef30 legacy=0x7fbd180af1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:10:42.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.699+0000 7fbd167fc640 10 monclient: _finish_hunting 0 2024-03-29T11:10:42.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.699+0000 7fbd167fc640 1 monclient: found mon.noname-c 2024-03-29T11:10:42.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.699+0000 7fbd167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:42.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.699+0000 7fbd167fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.70:6789/0 2024-03-29T11:10:42.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.699+0000 7fbd167fc640 1 -- 172.21.15.18:0/62260908 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd180ab490 con 0x7fbd1806d9d0 2024-03-29T11:10:42.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.699+0000 7fbd167fc640 10 monclient: _finish_auth 0 2024-03-29T11:10:42.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.699+0000 7fbd167fc640 10 monclient: _check_auth_tickets 2024-03-29T11:10:42.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.699+0000 7fbd167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:42.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.699+0000 7fbd167fc640 1 -- 172.21.15.18:0/62260908 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fbd0c001690 con 0x7fbd1806d9d0 2024-03-29T11:10:42.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.699+0000 7fbd167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:42.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.699+0000 7fbd167fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:10:42.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.699+0000 7fbd167fc640 10 monclient: dump: 2024-03-29T11:10:42.678 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:10:42.678 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:42.678 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:42.678 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:42.678 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:42.678 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:42.678 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:10:42.678 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:10:42.678 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:10:42.678 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:42.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.699+0000 7fbd1db04640 5 monclient: authenticate success, global_id 4291 2024-03-29T11:10:42.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.699+0000 7fbd1db04640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:10:42.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.700+0000 7fbd167fc640 1 -- 172.21.15.18:0/62260908 <== mon.1 v1:172.21.15.70:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd0c001860 con 0x7fbd1806d9d0 2024-03-29T11:10:42.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.700+0000 7fbd167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:42.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.700+0000 7fbd167fc640 1 -- 172.21.15.18:0/62260908 <== mon.1 v1:172.21.15.70:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fbd0c001cb0 con 0x7fbd1806d9d0 2024-03-29T11:10:42.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.700+0000 7fbd167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:42.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.700+0000 7fbd167fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:10:42.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.700+0000 7fbd167fc640 10 monclient: dump: 2024-03-29T11:10:42.680 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:10:42.680 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:42.680 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:42.680 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:42.680 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:42.681 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:42.681 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:10:42.681 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:10:42.681 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:10:42.681 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:42.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.700+0000 7fbd1db04640 10 monclient: get_monmap_and_config success 2024-03-29T11:10:42.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.700+0000 7fbd1db04640 4 set_mon_vals no callback set 2024-03-29T11:10:42.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.701+0000 7fbd1db04640 10 monclient: shutdown 2024-03-29T11:10:42.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.701+0000 7fbd1db04640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:42.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.701+0000 7fbd1db04640 1 -- 172.21.15.18:0/62260908 >> v1:172.21.15.70:6789/0 conn(0x7fbd1806d9d0 legacy=0x7fbd1813e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:42.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.703+0000 7fbd1db04640 1 -- 172.21.15.18:0/62260908 shutdown_connections 2024-03-29T11:10:42.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.703+0000 7fbd1db04640 1 -- 172.21.15.18:0/62260908 >> 172.21.15.18:0/62260908 conn(0x7fbd1806b660 msgr2=0x7fbd1806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:42.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.704+0000 7fbd1db04640 1 -- 172.21.15.18:0/62260908 shutdown_connections 2024-03-29T11:10:42.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.704+0000 7fbd1db04640 1 -- 172.21.15.18:0/62260908 wait complete. 2024-03-29T11:10:42.688 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.704+0000 7fbd1db04640 10 monclient: build_initial_monmap 2024-03-29T11:10:42.688 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.704+0000 7fbd1db04640 10 monclient: monmap: 2024-03-29T11:10:42.688 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:10:42.688 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:10:42.688 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:10:42.688 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:10:42.689 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:42.689 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:42.689 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:10:42.689 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:10:42.689 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:10:42.689 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:42.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.705+0000 7fbd1db04640 1 librados: starting msgr at 2024-03-29T11:10:42.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.705+0000 7fbd1db04640 1 librados: starting objecter 2024-03-29T11:10:42.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.705+0000 7fbd1db04640 1 Processor -- start 2024-03-29T11:10:42.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.705+0000 7fbd1db04640 1 -- start start 2024-03-29T11:10:42.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.705+0000 7fbd1db04640 1 librados: setting wanted keys 2024-03-29T11:10:42.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.705+0000 7fbd1db04640 1 librados: calling monclient init 2024-03-29T11:10:42.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.705+0000 7fbd1db04640 10 monclient: init 2024-03-29T11:10:42.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.706+0000 7fbd1db04640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:42.690 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.706+0000 7fbd1db04640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:10:42.690 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.708+0000 7fbd1db04640 10 monclient(hunting): picked mon.noname-c con 0x7fbd1806d9d0 addr v1:172.21.15.18:6790/0 2024-03-29T11:10:42.690 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.708+0000 7fbd1db04640 10 monclient(hunting): picked mon.noname-a con 0x7fbd180b07d0 addr v1:172.21.15.18:6789/0 2024-03-29T11:10:42.690 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.708+0000 7fbd1db04640 10 monclient(hunting): picked mon.noname-b con 0x7fbd1813ef30 addr v1:172.21.15.70:6789/0 2024-03-29T11:10:42.690 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.708+0000 7fbd1db04640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd1806f640 con 0x7fbd180b07d0 2024-03-29T11:10:42.690 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.708+0000 7fbd1db04640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd181e4af0 con 0x7fbd1813ef30 2024-03-29T11:10:42.690 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.708+0000 7fbd1db04640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd181e4c80 con 0x7fbd1806d9d0 2024-03-29T11:10:42.690 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.708+0000 7fbd1db04640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:42.690 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.709+0000 7fbd1db04640 10 monclient(hunting): authenticate will time out at 868.980918s 2024-03-29T11:10:42.690 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.709+0000 7fbd177fe640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fbd1806d9d0 0x7fbd181e43e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:45745/0 (socket says 172.21.15.18:45745) 2024-03-29T11:10:42.690 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.709+0000 7fbd177fe640 1 -- 172.21.15.18:0/615903357 learned_addr learned my addr 172.21.15.18:0/615903357 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:10:42.690 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.709+0000 7fbcfffff640 10 client.?.objecter ms_handle_connect 0x7fbd1806d9d0 2024-03-29T11:10:42.690 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.709+0000 7fbcfffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:42.690 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.709+0000 7fbcfffff640 1 -- 172.21.15.18:0/615903357 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3090412291 0 0) 0x7fbd180ab490 con 0x7fbd1806d9d0 2024-03-29T11:10:42.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.709+0000 7fbcfffff640 10 monclient(hunting): _negotiate 2024-03-29T11:10:42.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.709+0000 7fbcfffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:42.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.709+0000 7fbcfffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:42.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.709+0000 7fbcfffff640 10 monclient(hunting): my global_id is 4298 2024-03-29T11:10:42.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.709+0000 7fbcfffff640 1 -- 172.21.15.18:0/615903357 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd04004eb0 con 0x7fbd1806d9d0 2024-03-29T11:10:42.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.709+0000 7fbcfffff640 1 -- 172.21.15.18:0/615903357 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1018599331 0 0) 0x7fbd04004eb0 con 0x7fbd1806d9d0 2024-03-29T11:10:42.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.709+0000 7fbcfffff640 1 -- 172.21.15.18:0/615903357 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd180ab490 con 0x7fbd1806d9d0 2024-03-29T11:10:42.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.709+0000 7fbcfffff640 1 -- 172.21.15.18:0/615903357 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fbd0c001a90 con 0x7fbd1806d9d0 2024-03-29T11:10:42.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.709+0000 7fbcfffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:42.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.709+0000 7fbcfffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:10:42.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.709+0000 7fbcfffff640 10 monclient(hunting): dump: 2024-03-29T11:10:42.691 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:10:42.691 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:42.691 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:42.692 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:42.692 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:42.692 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:42.692 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:10:42.692 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:10:42.692 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:10:42.692 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:42.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.709+0000 7fbcfffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:10:42.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.710+0000 7fbcfffff640 1 -- 172.21.15.18:0/615903357 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 912040115 0 0) 0x7fbd180ab490 con 0x7fbd1806d9d0 2024-03-29T11:10:42.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.710+0000 7fbcfffff640 1 -- 172.21.15.18:0/615903357 >> v1:172.21.15.70:6789/0 conn(0x7fbd1813ef30 legacy=0x7fbd181ef5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:42.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.710+0000 7fbcfffff640 1 -- 172.21.15.18:0/615903357 >> v1:172.21.15.18:6789/0 conn(0x7fbd180b07d0 legacy=0x7fbd181e0f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:10:42.693 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.712+0000 7fbcfffff640 10 monclient: _finish_hunting 0 2024-03-29T11:10:42.693 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.712+0000 7fbcfffff640 1 monclient: found mon.c 2024-03-29T11:10:42.693 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.712+0000 7fbcfffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:42.693 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.712+0000 7fbcfffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:10:42.693 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.712+0000 7fbcfffff640 1 -- 172.21.15.18:0/615903357 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd181386b0 con 0x7fbd1806d9d0 2024-03-29T11:10:42.693 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.712+0000 7fbcfffff640 10 monclient: _finish_auth 0 2024-03-29T11:10:42.693 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.712+0000 7fbcfffff640 10 monclient: _check_auth_tickets 2024-03-29T11:10:42.693 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.712+0000 7fbcfffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:42.693 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.712+0000 7fbd1db04640 5 monclient: authenticate success, global_id 4298 2024-03-29T11:10:42.694 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.712+0000 7fbd1db04640 10 monclient: _renew_subs 2024-03-29T11:10:42.694 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.712+0000 7fbd1db04640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:10:42.694 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.712+0000 7fbd1db04640 1 -- 172.21.15.18:0/615903357 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd18138900 con 0x7fbd1806d9d0 2024-03-29T11:10:42.694 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.712+0000 7fbd1db04640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:10:42.694 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.712+0000 7fbd1db04640 10 monclient: _renew_subs 2024-03-29T11:10:42.694 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.712+0000 7fbd1db04640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:10:42.694 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.712+0000 7fbd1db04640 1 -- 172.21.15.18:0/615903357 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd18135760 con 0x7fbd1806d9d0 2024-03-29T11:10:42.694 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.712+0000 7fbd1db04640 1 librados: init done 2024-03-29T11:10:42.694 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.713+0000 7fbcfffff640 1 -- 172.21.15.18:0/615903357 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd0c001cf0 con 0x7fbd1806d9d0 2024-03-29T11:10:42.694 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.713+0000 7fbcfffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:42.694 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.713+0000 7fbcfffff640 1 -- 172.21.15.18:0/615903357 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fbd0c004c30 con 0x7fbd1806d9d0 2024-03-29T11:10:42.694 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.713+0000 7fbcfffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:42.694 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.713+0000 7fbcfffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:10:42.694 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.713+0000 7fbcfffff640 10 monclient: dump: 2024-03-29T11:10:42.695 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:10:42.695 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:42.695 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:42.695 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:42.695 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:42.695 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:42.695 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:10:42.695 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:10:42.695 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:10:42.695 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:42.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.713+0000 7fbd15ffb640 4 set_mon_vals no callback set 2024-03-29T11:10:42.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.715+0000 7fbcfffff640 1 -- 172.21.15.18:0/615903357 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7fbd0c004d90 con 0x7fbd1806d9d0 2024-03-29T11:10:42.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.715+0000 7fbd1db04640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:42.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.715+0000 7fbd1db04640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:42.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.715+0000 7fbd1db04640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:10:42.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.715+0000 7fbd1db04640 1 -- 172.21.15.18:0/615903357 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcf8005080 con 0x7fbd1806d9d0 2024-03-29T11:10:42.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.715+0000 7fbcfffff640 1 -- 172.21.15.18:0/615903357 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7fbd0c0849e0 con 0x7fbd1806d9d0 2024-03-29T11:10:42.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.715+0000 7fbcfffff640 10 client.4298.objecter ms_dispatch 0x7fbd18000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:10:42.696 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.715+0000 7fbcfffff640 3 client.4298.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:10:42.697 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.715+0000 7fbcfffff640 3 client.4298.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:10:42.697 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.715+0000 7fbcfffff640 20 client.4298.objecter dump_active .. 0 homeless 2024-03-29T11:10:42.697 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.715+0000 7fbd16ffd640 10 monclient: get_auth_request con 0x7fbd0406dba0 auth_method 0 2024-03-29T11:10:42.697 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.716+0000 7fbcfffff640 10 client.4298.objecter ms_handle_connect 0x7fbd0406dba0 2024-03-29T11:10:42.702 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.724+0000 7fbcfffff640 1 -- 172.21.15.18:0/615903357 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbd0c0502c0 con 0x7fbd1806d9d0 2024-03-29T11:10:42.703 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.724+0000 7fbcfffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:42.703 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:42.724+0000 7fbcfffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:10:43.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:43.044+0000 7fbd1db04640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:43.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:43.044+0000 7fbd1db04640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:43.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:43.044+0000 7fbd1db04640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:10:43.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:43.044+0000 7fbd1db04640 1 -- 172.21.15.18:0/615903357 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbcf8005330 con 0x7fbd1806d9d0 2024-03-29T11:10:43.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:43.045+0000 7fbcfffff640 1 -- 172.21.15.18:0/615903357 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7fbd0c0521f0 con 0x7fbd1806d9d0 2024-03-29T11:10:43.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:43.045+0000 7fbcfffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:43.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:43.045+0000 7fbcfffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:10:43.024 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:10:43.024 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:10:43.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:43.048+0000 7fbd1db04640 10 librados: watch_flush enter 2024-03-29T11:10:43.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:43.048+0000 7fbd1db04640 10 librados: watch_flush exit 2024-03-29T11:10:43.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:43.048+0000 7fbd1db04640 20 client.4298.objecter shutdown clearing up homeless session... 2024-03-29T11:10:43.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:43.048+0000 7fbd1db04640 10 client.4298.objecter successfully canceled tick 2024-03-29T11:10:43.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:43.049+0000 7fbd1db04640 1 -- 172.21.15.18:0/615903357 >> v1:172.21.15.70:6816/3101871562 conn(0x7fbd0406dba0 legacy=0x7fbd04070060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:43.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:43.049+0000 7fbd1db04640 10 monclient: shutdown 2024-03-29T11:10:43.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:43.049+0000 7fbd1db04640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:43.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:43.049+0000 7fbd1db04640 1 -- 172.21.15.18:0/615903357 >> v1:172.21.15.18:6790/0 conn(0x7fbd1806d9d0 legacy=0x7fbd181e43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:43.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:43.050+0000 7fbd1db04640 1 -- 172.21.15.18:0/615903357 shutdown_connections 2024-03-29T11:10:43.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:43.050+0000 7fbd1db04640 1 -- 172.21.15.18:0/615903357 >> 172.21.15.18:0/615903357 conn(0x7fbd1806b660 msgr2=0x7fbd1813e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:43.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:43.050+0000 7fbd1db04640 1 -- 172.21.15.18:0/615903357 shutdown_connections 2024-03-29T11:10:43.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:43.050+0000 7fbd1db04640 1 -- 172.21.15.18:0/615903357 wait complete. 2024-03-29T11:10:43.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:43.050+0000 7fbd1db04640 1 librados: shutdown 2024-03-29T11:10:43.047 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:10:49.854 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:10:50.000 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.022+0000 7f632d53a640 10 monclient: get_monmap_and_config 2024-03-29T11:10:50.000 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.022+0000 7f632d53a640 10 monclient: build_initial_monmap 2024-03-29T11:10:50.000 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.022+0000 7f632d53a640 10 monclient: monmap: 2024-03-29T11:10:50.000 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:10:50.000 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:50.000 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:10:50.023018+0000 2024-03-29T11:10:50.000 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:10:50.023018+0000 2024-03-29T11:10:50.001 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:50.001 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:50.001 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:10:50.001 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:10:50.001 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:10:50.001 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:50.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.022+0000 7f632d53a640 1 Processor -- start 2024-03-29T11:10:50.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.022+0000 7f632d53a640 1 -- start start 2024-03-29T11:10:50.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.022+0000 7f632d53a640 10 monclient: init 2024-03-29T11:10:50.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.023+0000 7f632d53a640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:50.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.023+0000 7f632d53a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:10:50.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.023+0000 7f632d53a640 10 monclient(hunting): picked mon.noname-c con 0x7f63281457f0 addr v1:172.21.15.70:6789/0 2024-03-29T11:10:50.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.023+0000 7f632d53a640 10 monclient(hunting): picked mon.noname-b con 0x7f632813f650 addr v1:172.21.15.18:6790/0 2024-03-29T11:10:50.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.023+0000 7f632d53a640 10 monclient(hunting): picked mon.noname-a con 0x7f63280a9d70 addr v1:172.21.15.18:6789/0 2024-03-29T11:10:50.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.023+0000 7f632d53a640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63280aaef0 con 0x7f63280a9d70 2024-03-29T11:10:50.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.023+0000 7f632d53a640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6328145bd0 con 0x7f63281457f0 2024-03-29T11:10:50.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.023+0000 7f632d53a640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6328145eb0 con 0x7f632813f650 2024-03-29T11:10:50.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.023+0000 7f632d53a640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:50.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.023+0000 7f632d53a640 10 monclient(hunting): authenticate will time out at 876.295802s 2024-03-29T11:10:50.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.024+0000 7f6326ffd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f63281457f0 0x7f632813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.18:41114/0 (socket says 172.21.15.18:41114) 2024-03-29T11:10:50.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.024+0000 7f6326ffd640 1 -- 172.21.15.18:0/1107824814 learned_addr learned my addr 172.21.15.18:0/1107824814 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:10:50.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.025+0000 7f6325ffb640 1 -- 172.21.15.18:0/1107824814 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 341104168 0 0) 0x7f6328145bd0 con 0x7f63281457f0 2024-03-29T11:10:50.004 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.025+0000 7f6325ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:10:50.004 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.025+0000 7f6325ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:50.004 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.025+0000 7f6325ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:50.004 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.025+0000 7f6325ffb640 10 monclient(hunting): my global_id is 4294 2024-03-29T11:10:50.004 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.025+0000 7f6325ffb640 1 -- 172.21.15.18:0/1107824814 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6310003520 con 0x7f63281457f0 2024-03-29T11:10:50.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.028+0000 7f6325ffb640 1 -- 172.21.15.18:0/1107824814 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1503706592 0 0) 0x7f6328145eb0 con 0x7f632813f650 2024-03-29T11:10:50.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.028+0000 7f6325ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:10:50.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.028+0000 7f6325ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:50.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.028+0000 7f6325ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:50.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.028+0000 7f6325ffb640 10 monclient(hunting): my global_id is 4301 2024-03-29T11:10:50.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.028+0000 7f6325ffb640 1 -- 172.21.15.18:0/1107824814 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6328145bd0 con 0x7f632813f650 2024-03-29T11:10:50.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.028+0000 7f6325ffb640 1 -- 172.21.15.18:0/1107824814 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1226065335 0 0) 0x7f6310003520 con 0x7f63281457f0 2024-03-29T11:10:50.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.028+0000 7f6325ffb640 1 -- 172.21.15.18:0/1107824814 >> v1:172.21.15.18:6790/0 conn(0x7f632813f650 legacy=0x7f63280b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:50.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.029+0000 7f6325ffb640 1 -- 172.21.15.18:0/1107824814 >> v1:172.21.15.18:6789/0 conn(0x7f63280a9d70 legacy=0x7f632814d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:10:50.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.029+0000 7f6325ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:10:50.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.029+0000 7f6325ffb640 1 monclient: found mon.noname-c 2024-03-29T11:10:50.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.029+0000 7f6325ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:50.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.029+0000 7f6325ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.70:6789/0 2024-03-29T11:10:50.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.029+0000 7f6325ffb640 1 -- 172.21.15.18:0/1107824814 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6328146340 con 0x7f63281457f0 2024-03-29T11:10:50.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.029+0000 7f6325ffb640 10 monclient: _finish_auth 0 2024-03-29T11:10:50.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.029+0000 7f6325ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:10:50.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.029+0000 7f6325ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:50.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.029+0000 7f6325ffb640 1 -- 172.21.15.18:0/1107824814 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f6318001690 con 0x7f63281457f0 2024-03-29T11:10:50.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.029+0000 7f6325ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:50.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.029+0000 7f6325ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:10:50.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.029+0000 7f6325ffb640 10 monclient: dump: 2024-03-29T11:10:50.010 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:10:50.010 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:50.010 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:50.010 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:50.010 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:50.010 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:50.010 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:10:50.010 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:10:50.011 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:10:50.011 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:50.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.030+0000 7f632d53a640 5 monclient: authenticate success, global_id 4294 2024-03-29T11:10:50.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.030+0000 7f632d53a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:10:50.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.033+0000 7f6325ffb640 1 -- 172.21.15.18:0/1107824814 <== mon.1 v1:172.21.15.70:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63180017f0 con 0x7f63281457f0 2024-03-29T11:10:50.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.033+0000 7f6325ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:50.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.033+0000 7f6325ffb640 1 -- 172.21.15.18:0/1107824814 <== mon.1 v1:172.21.15.70:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f6318001c00 con 0x7f63281457f0 2024-03-29T11:10:50.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.033+0000 7f6325ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:50.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.033+0000 7f6325ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:10:50.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.033+0000 7f6325ffb640 10 monclient: dump: 2024-03-29T11:10:50.012 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:10:50.012 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:50.012 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:50.012 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:50.012 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:50.012 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:50.013 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:10:50.013 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:10:50.013 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:10:50.013 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:50.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.033+0000 7f632d53a640 10 monclient: get_monmap_and_config success 2024-03-29T11:10:50.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.033+0000 7f632d53a640 4 set_mon_vals no callback set 2024-03-29T11:10:50.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.034+0000 7f632d53a640 10 monclient: shutdown 2024-03-29T11:10:50.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.034+0000 7f632d53a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:50.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.034+0000 7f632d53a640 1 -- 172.21.15.18:0/1107824814 >> v1:172.21.15.70:6789/0 conn(0x7f63281457f0 legacy=0x7f632813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:50.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.035+0000 7f632d53a640 1 -- 172.21.15.18:0/1107824814 shutdown_connections 2024-03-29T11:10:50.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.035+0000 7f632d53a640 1 -- 172.21.15.18:0/1107824814 >> 172.21.15.18:0/1107824814 conn(0x7f6328066a00 msgr2=0x7f6328066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:50.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.037+0000 7f632d53a640 1 -- 172.21.15.18:0/1107824814 shutdown_connections 2024-03-29T11:10:50.016 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.038+0000 7f632d53a640 1 -- 172.21.15.18:0/1107824814 wait complete. 2024-03-29T11:10:50.017 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.039+0000 7f632d53a640 10 monclient: build_initial_monmap 2024-03-29T11:10:50.017 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.039+0000 7f632d53a640 10 monclient: monmap: 2024-03-29T11:10:50.017 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:10:50.017 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:10:50.017 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:10:50.017 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:10:50.017 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:50.017 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:50.017 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:10:50.017 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:10:50.017 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:10:50.017 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:50.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.039+0000 7f632d53a640 1 librados: starting msgr at 2024-03-29T11:10:50.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.039+0000 7f632d53a640 1 librados: starting objecter 2024-03-29T11:10:50.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.040+0000 7f632d53a640 1 Processor -- start 2024-03-29T11:10:50.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.040+0000 7f632d53a640 1 -- start start 2024-03-29T11:10:50.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.040+0000 7f632d53a640 1 librados: setting wanted keys 2024-03-29T11:10:50.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.040+0000 7f632d53a640 1 librados: calling monclient init 2024-03-29T11:10:50.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.040+0000 7f632d53a640 10 monclient: init 2024-03-29T11:10:50.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.041+0000 7f632d53a640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:50.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.041+0000 7f632d53a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:10:50.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.041+0000 7f632d53a640 10 monclient(hunting): picked mon.noname-c con 0x7f63280a9d70 addr v1:172.21.15.18:6790/0 2024-03-29T11:10:50.019 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.041+0000 7f632d53a640 10 monclient(hunting): picked mon.noname-b con 0x7f632813f650 addr v1:172.21.15.70:6789/0 2024-03-29T11:10:50.020 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.041+0000 7f632d53a640 10 monclient(hunting): picked mon.noname-a con 0x7f63281457f0 addr v1:172.21.15.18:6789/0 2024-03-29T11:10:50.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.041+0000 7f632d53a640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f632806efa0 con 0x7f63281457f0 2024-03-29T11:10:50.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.041+0000 7f632d53a640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f632806f210 con 0x7f632813f650 2024-03-29T11:10:50.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.041+0000 7f632d53a640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63281381a0 con 0x7f63280a9d70 2024-03-29T11:10:50.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.041+0000 7f632d53a640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:50.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.041+0000 7f632d53a640 10 monclient(hunting): authenticate will time out at 876.313139s 2024-03-29T11:10:50.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.041+0000 7f63267fc640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f632813f650 0x7f6328138d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.18:41452/0 (socket says 172.21.15.18:41452) 2024-03-29T11:10:50.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.041+0000 7f63267fc640 1 -- 172.21.15.18:0/2114177951 learned_addr learned my addr 172.21.15.18:0/2114177951 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:10:50.031 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.046+0000 7f630f7fe640 10 client.?.objecter ms_handle_connect 0x7f632813f650 2024-03-29T11:10:50.031 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.046+0000 7f630f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:50.031 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.046+0000 7f630f7fe640 10 client.?.objecter ms_handle_connect 0x7f63280a9d70 2024-03-29T11:10:50.031 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.046+0000 7f630f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:50.031 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.046+0000 7f630f7fe640 10 client.?.objecter ms_handle_connect 0x7f63281457f0 2024-03-29T11:10:50.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.046+0000 7f630f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:50.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.046+0000 7f630f7fe640 1 -- 172.21.15.18:0/2114177951 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 119624983 0 0) 0x7f63281381a0 con 0x7f63280a9d70 2024-03-29T11:10:50.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.046+0000 7f630f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:10:50.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.046+0000 7f630f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:50.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.046+0000 7f630f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:50.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.046+0000 7f630f7fe640 10 monclient(hunting): my global_id is 4304 2024-03-29T11:10:50.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.046+0000 7f630f7fe640 1 -- 172.21.15.18:0/2114177951 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6314004eb0 con 0x7f63280a9d70 2024-03-29T11:10:50.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.046+0000 7f630f7fe640 1 -- 172.21.15.18:0/2114177951 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 903268618 0 0) 0x7f632806efa0 con 0x7f63281457f0 2024-03-29T11:10:50.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.046+0000 7f630f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:10:50.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.046+0000 7f630f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:50.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.046+0000 7f630f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:50.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.046+0000 7f630f7fe640 10 monclient(hunting): my global_id is 4287 2024-03-29T11:10:50.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.046+0000 7f630f7fe640 1 -- 172.21.15.18:0/2114177951 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63281381a0 con 0x7f63281457f0 2024-03-29T11:10:50.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.047+0000 7f630f7fe640 1 -- 172.21.15.18:0/2114177951 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3086711289 0 0) 0x7f63281381a0 con 0x7f63281457f0 2024-03-29T11:10:50.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.047+0000 7f630f7fe640 1 -- 172.21.15.18:0/2114177951 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f632806efa0 con 0x7f63281457f0 2024-03-29T11:10:50.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.047+0000 7f630f7fe640 1 -- 172.21.15.18:0/2114177951 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 695414997 0 0) 0x7f6314004eb0 con 0x7f63280a9d70 2024-03-29T11:10:50.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.047+0000 7f630f7fe640 1 -- 172.21.15.18:0/2114177951 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63281381a0 con 0x7f63280a9d70 2024-03-29T11:10:50.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.047+0000 7f630f7fe640 1 -- 172.21.15.18:0/2114177951 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2820168190 0 0) 0x7f632806f210 con 0x7f632813f650 2024-03-29T11:10:50.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.047+0000 7f630f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:10:50.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.047+0000 7f630f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:50.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.047+0000 7f630f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:50.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.047+0000 7f630f7fe640 10 monclient(hunting): my global_id is 4297 2024-03-29T11:10:50.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.047+0000 7f630f7fe640 1 -- 172.21.15.18:0/2114177951 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6314004eb0 con 0x7f632813f650 2024-03-29T11:10:50.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.047+0000 7f630f7fe640 1 -- 172.21.15.18:0/2114177951 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f6320001b10 con 0x7f63281457f0 2024-03-29T11:10:50.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.047+0000 7f630f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:50.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.047+0000 7f630f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:10:50.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.047+0000 7f630f7fe640 10 monclient(hunting): dump: 2024-03-29T11:10:50.033 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:10:50.034 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:50.034 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:50.034 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:50.034 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:50.034 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:50.034 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:10:50.034 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:10:50.034 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:10:50.034 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:50.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.048+0000 7f630f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:10:50.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.048+0000 7f630f7fe640 1 -- 172.21.15.18:0/2114177951 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f6318001b00 con 0x7f63280a9d70 2024-03-29T11:10:50.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.048+0000 7f630f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:50.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.048+0000 7f630f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:10:50.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.048+0000 7f630f7fe640 10 monclient(hunting): dump: 2024-03-29T11:10:50.035 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:10:50.035 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:50.035 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:50.035 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:50.035 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:50.035 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:50.035 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:10:50.035 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:10:50.035 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:10:50.035 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:50.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.048+0000 7f630f7fe640 1 -- 172.21.15.18:0/2114177951 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2711261301 0 0) 0x7f63281381a0 con 0x7f63280a9d70 2024-03-29T11:10:50.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.048+0000 7f630f7fe640 1 -- 172.21.15.18:0/2114177951 >> v1:172.21.15.70:6789/0 conn(0x7f632813f650 legacy=0x7f6328138d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:50.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.048+0000 7f630f7fe640 1 -- 172.21.15.18:0/2114177951 >> v1:172.21.15.18:6789/0 conn(0x7f63281457f0 legacy=0x7f6328139460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:50.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.048+0000 7f630f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:10:50.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.048+0000 7f630f7fe640 1 monclient: found mon.c 2024-03-29T11:10:50.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.048+0000 7f630f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:50.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.048+0000 7f630f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:10:50.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.048+0000 7f630f7fe640 1 -- 172.21.15.18:0/2114177951 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63281ded60 con 0x7f63280a9d70 2024-03-29T11:10:50.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.048+0000 7f630f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:10:50.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.048+0000 7f630f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:10:50.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.048+0000 7f630f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:50.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.049+0000 7f632d53a640 5 monclient: authenticate success, global_id 4304 2024-03-29T11:10:50.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.049+0000 7f632d53a640 10 monclient: _renew_subs 2024-03-29T11:10:50.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.049+0000 7f632d53a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:10:50.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.049+0000 7f632d53a640 1 -- 172.21.15.18:0/2114177951 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6328139b70 con 0x7f63280a9d70 2024-03-29T11:10:50.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.049+0000 7f632d53a640 10 client.4304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:10:50.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.049+0000 7f632d53a640 10 monclient: _renew_subs 2024-03-29T11:10:50.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.049+0000 7f632d53a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:10:50.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.049+0000 7f632d53a640 1 -- 172.21.15.18:0/2114177951 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f632813abf0 con 0x7f63280a9d70 2024-03-29T11:10:50.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.049+0000 7f632d53a640 1 librados: init done 2024-03-29T11:10:50.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.050+0000 7f630f7fe640 1 -- 172.21.15.18:0/2114177951 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63180010a0 con 0x7f63280a9d70 2024-03-29T11:10:50.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.050+0000 7f630f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:50.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.050+0000 7f630f7fe640 1 -- 172.21.15.18:0/2114177951 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f6318004a90 con 0x7f63280a9d70 2024-03-29T11:10:50.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.050+0000 7f630f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:50.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.050+0000 7f630f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:10:50.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.050+0000 7f630f7fe640 10 monclient: dump: 2024-03-29T11:10:50.037 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:10:50.037 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:50.037 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:50.037 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:50.037 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:50.038 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:50.038 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:10:50.038 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:10:50.038 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:10:50.038 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:50.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.050+0000 7f6324ff9640 4 set_mon_vals no callback set 2024-03-29T11:10:50.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.051+0000 7f632d53a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:50.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.051+0000 7f632d53a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:50.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.051+0000 7f632d53a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:10:50.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.051+0000 7f632d53a640 1 -- 172.21.15.18:0/2114177951 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6308005080 con 0x7f63280a9d70 2024-03-29T11:10:50.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.052+0000 7f630f7fe640 1 -- 172.21.15.18:0/2114177951 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f631801aa60 con 0x7f63280a9d70 2024-03-29T11:10:50.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.052+0000 7f630f7fe640 1 -- 172.21.15.18:0/2114177951 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f6318084930 con 0x7f63280a9d70 2024-03-29T11:10:50.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.052+0000 7f630f7fe640 10 client.4304.objecter ms_dispatch 0x7f6328000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:10:50.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.052+0000 7f630f7fe640 3 client.4304.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:10:50.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.052+0000 7f630f7fe640 3 client.4304.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:10:50.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.052+0000 7f630f7fe640 20 client.4304.objecter dump_active .. 0 homeless 2024-03-29T11:10:50.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.055+0000 7f63267fc640 10 monclient: get_auth_request con 0x7f631406df80 auth_method 0 2024-03-29T11:10:50.040 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.056+0000 7f630f7fe640 10 client.4304.objecter ms_handle_connect 0x7f631406df80 2024-03-29T11:10:50.040 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.061+0000 7f630f7fe640 1 -- 172.21.15.18:0/2114177951 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6318050040 con 0x7f63280a9d70 2024-03-29T11:10:50.040 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.061+0000 7f630f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:50.040 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.061+0000 7f630f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:10:50.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.389+0000 7f632d53a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:50.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.389+0000 7f632d53a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:50.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.389+0000 7f632d53a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:10:50.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.389+0000 7f632d53a640 1 -- 172.21.15.18:0/2114177951 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6308005330 con 0x7f63280a9d70 2024-03-29T11:10:50.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.390+0000 7f630f7fe640 1 -- 172.21.15.18:0/2114177951 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f6318052140 con 0x7f63280a9d70 2024-03-29T11:10:50.369 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.390+0000 7f630f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:50.369 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.391+0000 7f630f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:10:50.369 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:10:50.370 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:10:50.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.395+0000 7f632d53a640 10 librados: watch_flush enter 2024-03-29T11:10:50.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.395+0000 7f632d53a640 10 librados: watch_flush exit 2024-03-29T11:10:50.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.395+0000 7f632d53a640 20 client.4304.objecter shutdown clearing up homeless session... 2024-03-29T11:10:50.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.395+0000 7f632d53a640 10 client.4304.objecter successfully canceled tick 2024-03-29T11:10:50.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.395+0000 7f632d53a640 1 -- 172.21.15.18:0/2114177951 >> v1:172.21.15.70:6816/3101871562 conn(0x7f631406df80 legacy=0x7f6314070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:50.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.395+0000 7f632d53a640 10 monclient: shutdown 2024-03-29T11:10:50.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.395+0000 7f632d53a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:50.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.395+0000 7f632d53a640 1 -- 172.21.15.18:0/2114177951 >> v1:172.21.15.18:6790/0 conn(0x7f63280a9d70 legacy=0x7f63281df280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:50.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.395+0000 7f632d53a640 1 -- 172.21.15.18:0/2114177951 shutdown_connections 2024-03-29T11:10:50.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.395+0000 7f632d53a640 1 -- 172.21.15.18:0/2114177951 >> 172.21.15.18:0/2114177951 conn(0x7f6328066a00 msgr2=0x7f6328066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:50.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.395+0000 7f632d53a640 1 -- 172.21.15.18:0/2114177951 shutdown_connections 2024-03-29T11:10:50.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.395+0000 7f632d53a640 1 -- 172.21.15.18:0/2114177951 wait complete. 2024-03-29T11:10:50.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:50.396+0000 7f632d53a640 1 librados: shutdown 2024-03-29T11:10:50.394 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:10:57.201 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:10:57.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.373+0000 7fb7d2eff640 10 monclient: get_monmap_and_config 2024-03-29T11:10:57.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.373+0000 7fb7d2eff640 10 monclient: build_initial_monmap 2024-03-29T11:10:57.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.373+0000 7fb7d2eff640 10 monclient: monmap: 2024-03-29T11:10:57.353 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:10:57.353 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:57.353 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:10:57.375286+0000 2024-03-29T11:10:57.353 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:10:57.375286+0000 2024-03-29T11:10:57.353 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:57.353 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:57.353 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:10:57.353 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:10:57.353 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:10:57.353 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:57.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.377+0000 7fb7d2eff640 1 Processor -- start 2024-03-29T11:10:57.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.377+0000 7fb7d2eff640 1 -- start start 2024-03-29T11:10:57.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.377+0000 7fb7d2eff640 10 monclient: init 2024-03-29T11:10:57.358 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.378+0000 7fb7d2eff640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:57.358 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.378+0000 7fb7d2eff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:10:57.358 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.378+0000 7fb7d2eff640 10 monclient(hunting): picked mon.noname-c con 0x7fb7cc1457f0 addr v1:172.21.15.70:6789/0 2024-03-29T11:10:57.358 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.378+0000 7fb7d2eff640 10 monclient(hunting): picked mon.noname-a con 0x7fb7cc13f650 addr v1:172.21.15.18:6789/0 2024-03-29T11:10:57.358 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.378+0000 7fb7d2eff640 10 monclient(hunting): picked mon.noname-b con 0x7fb7cc0a9d70 addr v1:172.21.15.18:6790/0 2024-03-29T11:10:57.358 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.379+0000 7fb7d2eff640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7cc0aaef0 con 0x7fb7cc13f650 2024-03-29T11:10:57.358 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.379+0000 7fb7d2eff640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7cc145bd0 con 0x7fb7cc1457f0 2024-03-29T11:10:57.358 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.379+0000 7fb7d2eff640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7cc145eb0 con 0x7fb7cc0a9d70 2024-03-29T11:10:57.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.379+0000 7fb7d2eff640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:57.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.379+0000 7fb7d2eff640 10 monclient(hunting): authenticate will time out at 883.652198s 2024-03-29T11:10:57.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.381+0000 7fb7d0c9b640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fb7cc1457f0 0x7fb7cc13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.18:44203/0 (socket says 172.21.15.18:44203) 2024-03-29T11:10:57.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.381+0000 7fb7d0c9b640 1 -- 172.21.15.18:0/3310006634 learned_addr learned my addr 172.21.15.18:0/3310006634 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:10:57.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.382+0000 7fb7cb7fe640 1 -- 172.21.15.18:0/3310006634 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1967116817 0 0) 0x7fb7cc145bd0 con 0x7fb7cc1457f0 2024-03-29T11:10:57.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.382+0000 7fb7cb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:10:57.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.382+0000 7fb7cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:57.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.382+0000 7fb7cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:57.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.382+0000 7fb7cb7fe640 10 monclient(hunting): my global_id is 4300 2024-03-29T11:10:57.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.382+0000 7fb7cb7fe640 1 -- 172.21.15.18:0/3310006634 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7b4003520 con 0x7fb7cc1457f0 2024-03-29T11:10:57.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.383+0000 7fb7cb7fe640 1 -- 172.21.15.18:0/3310006634 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3847821047 0 0) 0x7fb7cc145eb0 con 0x7fb7cc0a9d70 2024-03-29T11:10:57.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.383+0000 7fb7cb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:10:57.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.383+0000 7fb7cb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:57.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.383+0000 7fb7cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:57.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.383+0000 7fb7cb7fe640 10 monclient(hunting): my global_id is 4307 2024-03-29T11:10:57.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.383+0000 7fb7cb7fe640 1 -- 172.21.15.18:0/3310006634 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7cc145bd0 con 0x7fb7cc0a9d70 2024-03-29T11:10:57.365 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.386+0000 7fb7cb7fe640 1 -- 172.21.15.18:0/3310006634 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2921681944 0 0) 0x7fb7b4003520 con 0x7fb7cc1457f0 2024-03-29T11:10:57.365 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.386+0000 7fb7cb7fe640 1 -- 172.21.15.18:0/3310006634 >> v1:172.21.15.18:6790/0 conn(0x7fb7cc0a9d70 legacy=0x7fb7cc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:57.365 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.386+0000 7fb7cb7fe640 1 -- 172.21.15.18:0/3310006634 >> v1:172.21.15.18:6789/0 conn(0x7fb7cc13f650 legacy=0x7fb7cc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:10:57.365 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.386+0000 7fb7cb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:10:57.366 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.386+0000 7fb7cb7fe640 1 monclient: found mon.noname-c 2024-03-29T11:10:57.366 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.386+0000 7fb7cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:57.366 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.386+0000 7fb7cb7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.70:6789/0 2024-03-29T11:10:57.366 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.386+0000 7fb7cb7fe640 1 -- 172.21.15.18:0/3310006634 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7cc146340 con 0x7fb7cc1457f0 2024-03-29T11:10:57.366 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.386+0000 7fb7cb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:10:57.366 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.386+0000 7fb7cb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:10:57.366 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.386+0000 7fb7cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:57.366 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.386+0000 7fb7cb7fe640 1 -- 172.21.15.18:0/3310006634 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fb7c4001a10 con 0x7fb7cc1457f0 2024-03-29T11:10:57.366 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.386+0000 7fb7cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:57.366 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.386+0000 7fb7cb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:10:57.366 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.386+0000 7fb7cb7fe640 10 monclient: dump: 2024-03-29T11:10:57.366 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:10:57.366 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:57.366 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:57.367 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:57.367 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:57.367 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:57.367 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:10:57.367 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:10:57.367 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:10:57.367 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:57.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.386+0000 7fb7cb7fe640 1 -- 172.21.15.18:0/3310006634 <== mon.1 v1:172.21.15.70:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7c4001b70 con 0x7fb7cc1457f0 2024-03-29T11:10:57.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.386+0000 7fb7cb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:57.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.386+0000 7fb7cb7fe640 1 -- 172.21.15.18:0/3310006634 <== mon.1 v1:172.21.15.70:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fb7c4001f80 con 0x7fb7cc1457f0 2024-03-29T11:10:57.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.386+0000 7fb7cb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:57.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.386+0000 7fb7cb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:10:57.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.386+0000 7fb7cb7fe640 10 monclient: dump: 2024-03-29T11:10:57.367 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:10:57.368 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:57.368 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:57.368 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:57.368 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:57.368 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:57.368 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:10:57.368 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:10:57.368 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:10:57.368 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:57.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.386+0000 7fb7d2eff640 5 monclient: authenticate success, global_id 4300 2024-03-29T11:10:57.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.386+0000 7fb7d2eff640 10 monclient: get_monmap_and_config success 2024-03-29T11:10:57.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.386+0000 7fb7d2eff640 4 set_mon_vals no callback set 2024-03-29T11:10:57.369 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.387+0000 7fb7d2eff640 10 monclient: shutdown 2024-03-29T11:10:57.369 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.387+0000 7fb7d2eff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:57.369 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.387+0000 7fb7d2eff640 1 -- 172.21.15.18:0/3310006634 >> v1:172.21.15.70:6789/0 conn(0x7fb7cc1457f0 legacy=0x7fb7cc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:57.369 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.387+0000 7fb7d2eff640 1 -- 172.21.15.18:0/3310006634 shutdown_connections 2024-03-29T11:10:57.369 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.387+0000 7fb7d2eff640 1 -- 172.21.15.18:0/3310006634 >> 172.21.15.18:0/3310006634 conn(0x7fb7cc066a00 msgr2=0x7fb7cc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:57.369 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.388+0000 7fb7d2eff640 1 -- 172.21.15.18:0/3310006634 shutdown_connections 2024-03-29T11:10:57.370 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.391+0000 7fb7d2eff640 1 -- 172.21.15.18:0/3310006634 wait complete. 2024-03-29T11:10:57.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.392+0000 7fb7d2eff640 10 monclient: build_initial_monmap 2024-03-29T11:10:57.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.392+0000 7fb7d2eff640 10 monclient: monmap: 2024-03-29T11:10:57.373 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:10:57.373 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:10:57.373 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:10:57.373 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:10:57.373 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:10:57.373 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:57.373 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:10:57.373 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:10:57.373 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:10:57.373 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:57.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.393+0000 7fb7d2eff640 1 librados: starting msgr at 2024-03-29T11:10:57.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.393+0000 7fb7d2eff640 1 librados: starting objecter 2024-03-29T11:10:57.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.393+0000 7fb7d2eff640 1 Processor -- start 2024-03-29T11:10:57.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.393+0000 7fb7d2eff640 1 -- start start 2024-03-29T11:10:57.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.393+0000 7fb7d2eff640 1 librados: setting wanted keys 2024-03-29T11:10:57.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.393+0000 7fb7d2eff640 1 librados: calling monclient init 2024-03-29T11:10:57.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.393+0000 7fb7d2eff640 10 monclient: init 2024-03-29T11:10:57.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.394+0000 7fb7d2eff640 10 monclient: _reopen_session rank -1 2024-03-29T11:10:57.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.395+0000 7fb7d2eff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:10:57.377 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.398+0000 7fb7d2eff640 10 monclient(hunting): picked mon.noname-a con 0x7fb7cc0a9d70 addr v1:172.21.15.18:6789/0 2024-03-29T11:10:57.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.399+0000 7fb7d2eff640 10 monclient(hunting): picked mon.noname-b con 0x7fb7cc13f650 addr v1:172.21.15.70:6789/0 2024-03-29T11:10:57.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.399+0000 7fb7d2eff640 10 monclient(hunting): picked mon.noname-c con 0x7fb7cc1457f0 addr v1:172.21.15.18:6790/0 2024-03-29T11:10:57.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.399+0000 7fb7d2eff640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7cc1420a0 con 0x7fb7cc0a9d70 2024-03-29T11:10:57.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.399+0000 7fb7d2eff640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7cc142230 con 0x7fb7cc13f650 2024-03-29T11:10:57.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.399+0000 7fb7d2eff640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7cc1423c0 con 0x7fb7cc1457f0 2024-03-29T11:10:57.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.399+0000 7fb7d2eff640 10 monclient(hunting): _renew_subs 2024-03-29T11:10:57.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.399+0000 7fb7d2eff640 10 monclient(hunting): authenticate will time out at 883.672102s 2024-03-29T11:10:57.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.399+0000 7fb7d0c9b640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fb7cc0a9d70 0x7fb7cc0b0d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:47747/0 (socket says 172.21.15.18:47747) 2024-03-29T11:10:57.379 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.399+0000 7fb7d0c9b640 1 -- 172.21.15.18:0/2180833409 learned_addr learned my addr 172.21.15.18:0/2180833409 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:10:57.379 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.399+0000 7fb7c97fa640 10 client.?.objecter ms_handle_connect 0x7fb7cc0a9d70 2024-03-29T11:10:57.379 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.399+0000 7fb7c97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:10:57.379 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.400+0000 7fb7c97fa640 1 -- 172.21.15.18:0/2180833409 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4116326992 0 0) 0x7fb7cc1420a0 con 0x7fb7cc0a9d70 2024-03-29T11:10:57.379 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.400+0000 7fb7c97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:10:57.379 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.400+0000 7fb7c97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:10:57.379 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.400+0000 7fb7c97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:10:57.380 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.400+0000 7fb7c97fa640 10 monclient(hunting): my global_id is 4290 2024-03-29T11:10:57.380 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.400+0000 7fb7c97fa640 1 -- 172.21.15.18:0/2180833409 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7b8004eb0 con 0x7fb7cc0a9d70 2024-03-29T11:10:57.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.402+0000 7fb7c97fa640 1 -- 172.21.15.18:0/2180833409 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3987099080 0 0) 0x7fb7b8004eb0 con 0x7fb7cc0a9d70 2024-03-29T11:10:57.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.402+0000 7fb7c97fa640 1 -- 172.21.15.18:0/2180833409 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7cc1420a0 con 0x7fb7cc0a9d70 2024-03-29T11:10:57.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.402+0000 7fb7c97fa640 1 -- 172.21.15.18:0/2180833409 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fb7c4001f40 con 0x7fb7cc0a9d70 2024-03-29T11:10:57.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.402+0000 7fb7c97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:57.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.402+0000 7fb7c97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:10:57.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.402+0000 7fb7c97fa640 10 monclient(hunting): dump: 2024-03-29T11:10:57.381 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:10:57.381 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:57.381 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:57.381 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:57.381 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:57.381 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:57.382 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:10:57.382 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:10:57.382 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:10:57.382 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:57.382 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.402+0000 7fb7c97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:10:57.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.403+0000 7fb7c97fa640 1 -- 172.21.15.18:0/2180833409 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1806598113 0 0) 0x7fb7cc1420a0 con 0x7fb7cc0a9d70 2024-03-29T11:10:57.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.403+0000 7fb7c97fa640 1 -- 172.21.15.18:0/2180833409 >> v1:172.21.15.18:6790/0 conn(0x7fb7cc1457f0 legacy=0x7fb7cc1d79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:57.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.403+0000 7fb7c97fa640 1 -- 172.21.15.18:0/2180833409 >> v1:172.21.15.70:6789/0 conn(0x7fb7cc13f650 legacy=0x7fb7cc1d72d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:10:57.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.403+0000 7fb7c97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:10:57.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.403+0000 7fb7c97fa640 1 monclient: found mon.a 2024-03-29T11:10:57.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.403+0000 7fb7c97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:10:57.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.403+0000 7fb7c97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:10:57.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.403+0000 7fb7c97fa640 1 -- 172.21.15.18:0/2180833409 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7cc0ad420 con 0x7fb7cc0a9d70 2024-03-29T11:10:57.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.403+0000 7fb7c97fa640 10 monclient: _finish_auth 0 2024-03-29T11:10:57.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.403+0000 7fb7c97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:10:57.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.403+0000 7fb7c97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:10:57.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.404+0000 7fb7c97fa640 1 -- 172.21.15.18:0/2180833409 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7c4004e10 con 0x7fb7cc0a9d70 2024-03-29T11:10:57.384 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.404+0000 7fb7c97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:10:57.384 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.404+0000 7fb7c97fa640 1 -- 172.21.15.18:0/2180833409 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fb7c4005120 con 0x7fb7cc0a9d70 2024-03-29T11:10:57.384 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.404+0000 7fb7c97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:10:57.384 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.404+0000 7fb7c97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:10:57.384 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.404+0000 7fb7c97fa640 10 monclient: dump: 2024-03-29T11:10:57.384 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:10:57.384 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:10:57.384 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:57.384 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:10:57.384 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:10:57.384 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:10:57.384 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:10:57.384 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:10:57.384 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:10:57.385 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:10:57.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.404+0000 7fb7d2eff640 5 monclient: authenticate success, global_id 4290 2024-03-29T11:10:57.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.404+0000 7fb7d2eff640 10 monclient: _renew_subs 2024-03-29T11:10:57.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.404+0000 7fb7d2eff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:10:57.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.404+0000 7fb7d2eff640 1 -- 172.21.15.18:0/2180833409 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7cc0ad670 con 0x7fb7cc0a9d70 2024-03-29T11:10:57.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.404+0000 7fb7d2eff640 10 client.4290.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:10:57.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.404+0000 7fb7d2eff640 10 monclient: _renew_subs 2024-03-29T11:10:57.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.404+0000 7fb7d2eff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:10:57.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.404+0000 7fb7d2eff640 1 -- 172.21.15.18:0/2180833409 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7cc0b1630 con 0x7fb7cc0a9d70 2024-03-29T11:10:57.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.404+0000 7fb7d2eff640 1 librados: init done 2024-03-29T11:10:57.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.404+0000 7fb7ca7fc640 4 set_mon_vals no callback set 2024-03-29T11:10:57.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.406+0000 7fb7d2eff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:57.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.406+0000 7fb7d2eff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:57.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.406+0000 7fb7d2eff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:10:57.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.406+0000 7fb7d2eff640 1 -- 172.21.15.18:0/2180833409 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7ac005080 con 0x7fb7cc0a9d70 2024-03-29T11:10:57.395 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.409+0000 7fb7c97fa640 1 -- 172.21.15.18:0/2180833409 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7fb7c401b0f0 con 0x7fb7cc0a9d70 2024-03-29T11:10:57.395 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.410+0000 7fb7c97fa640 1 -- 172.21.15.18:0/2180833409 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7fb7c4084eb0 con 0x7fb7cc0a9d70 2024-03-29T11:10:57.395 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.410+0000 7fb7c97fa640 10 client.4290.objecter ms_dispatch 0x7fb7cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:10:57.395 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.410+0000 7fb7c97fa640 3 client.4290.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:10:57.395 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.410+0000 7fb7c97fa640 3 client.4290.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:10:57.395 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.410+0000 7fb7c97fa640 20 client.4290.objecter dump_active .. 0 homeless 2024-03-29T11:10:57.395 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.416+0000 7fb7cbfff640 10 monclient: get_auth_request con 0x7fb7b806dc70 auth_method 0 2024-03-29T11:10:57.396 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.416+0000 7fb7c97fa640 10 client.4290.objecter ms_handle_connect 0x7fb7b806dc70 2024-03-29T11:10:57.396 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.416+0000 7fb7c97fa640 1 -- 172.21.15.18:0/2180833409 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb7c4050790 con 0x7fb7cc0a9d70 2024-03-29T11:10:57.396 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.416+0000 7fb7c97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:10:57.396 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.416+0000 7fb7c97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:10:57.717 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.737+0000 7fb7d2eff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:57.717 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.737+0000 7fb7d2eff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:57.717 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.737+0000 7fb7d2eff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:10:57.717 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.737+0000 7fb7d2eff640 1 -- 172.21.15.18:0/2180833409 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7ac005330 con 0x7fb7cc0a9d70 2024-03-29T11:10:57.719 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.740+0000 7fb7c97fa640 1 -- 172.21.15.18:0/2180833409 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7fb7c40526c0 con 0x7fb7cc0a9d70 2024-03-29T11:10:57.719 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.740+0000 7fb7c97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:10:57.719 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.740+0000 7fb7c97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:10:57.721 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:10:57.721 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:10:57.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.745+0000 7fb7d2eff640 10 librados: watch_flush enter 2024-03-29T11:10:57.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.745+0000 7fb7d2eff640 10 librados: watch_flush exit 2024-03-29T11:10:57.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.745+0000 7fb7d2eff640 20 client.4290.objecter shutdown clearing up homeless session... 2024-03-29T11:10:57.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.745+0000 7fb7d2eff640 10 client.4290.objecter successfully canceled tick 2024-03-29T11:10:57.727 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.746+0000 7fb7d2eff640 1 -- 172.21.15.18:0/2180833409 >> v1:172.21.15.70:6816/3101871562 conn(0x7fb7b806dc70 legacy=0x7fb7b8070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:57.727 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.746+0000 7fb7d2eff640 10 monclient: shutdown 2024-03-29T11:10:57.727 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.746+0000 7fb7d2eff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:10:57.727 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.746+0000 7fb7d2eff640 1 -- 172.21.15.18:0/2180833409 >> v1:172.21.15.18:6789/0 conn(0x7fb7cc0a9d70 legacy=0x7fb7cc0b0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:10:57.728 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.747+0000 7fb7d2eff640 1 -- 172.21.15.18:0/2180833409 shutdown_connections 2024-03-29T11:10:57.728 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.747+0000 7fb7d2eff640 1 -- 172.21.15.18:0/2180833409 >> 172.21.15.18:0/2180833409 conn(0x7fb7cc066a00 msgr2=0x7fb7cc0b4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:10:57.728 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.747+0000 7fb7d2eff640 1 -- 172.21.15.18:0/2180833409 shutdown_connections 2024-03-29T11:10:57.728 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.747+0000 7fb7d2eff640 1 -- 172.21.15.18:0/2180833409 wait complete. 2024-03-29T11:10:57.728 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:10:57.748+0000 7fb7d2eff640 1 librados: shutdown 2024-03-29T11:10:57.744 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:11:04.551 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:11:04.705 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.723+0000 7efe556e4640 10 monclient: get_monmap_and_config 2024-03-29T11:11:04.705 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.723+0000 7efe556e4640 10 monclient: build_initial_monmap 2024-03-29T11:11:04.705 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.723+0000 7efe556e4640 10 monclient: monmap: 2024-03-29T11:11:04.705 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:11:04.705 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:04.705 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:11:04.725815+0000 2024-03-29T11:11:04.706 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:11:04.725815+0000 2024-03-29T11:11:04.706 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:04.706 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:04.706 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:11:04.706 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:11:04.706 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:11:04.706 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:04.707 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.728+0000 7efe556e4640 1 Processor -- start 2024-03-29T11:11:04.708 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.728+0000 7efe556e4640 1 -- start start 2024-03-29T11:11:04.708 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.728+0000 7efe556e4640 10 monclient: init 2024-03-29T11:11:04.709 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.729+0000 7efe556e4640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:04.709 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.729+0000 7efe556e4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:11:04.709 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.729+0000 7efe556e4640 10 monclient(hunting): picked mon.noname-b con 0x7efe500b4100 addr v1:172.21.15.18:6790/0 2024-03-29T11:11:04.709 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.729+0000 7efe556e4640 10 monclient(hunting): picked mon.noname-a con 0x7efe5013ee90 addr v1:172.21.15.18:6789/0 2024-03-29T11:11:04.709 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.729+0000 7efe556e4640 10 monclient(hunting): picked mon.noname-c con 0x7efe500af900 addr v1:172.21.15.70:6789/0 2024-03-29T11:11:04.709 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.729+0000 7efe556e4640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe500afd10 con 0x7efe5013ee90 2024-03-29T11:11:04.710 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.729+0000 7efe556e4640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe500af6f0 con 0x7efe500af900 2024-03-29T11:11:04.710 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.729+0000 7efe556e4640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe500aac80 con 0x7efe500b4100 2024-03-29T11:11:04.710 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.729+0000 7efe556e4640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:04.710 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.729+0000 7efe556e4640 10 monclient(hunting): authenticate will time out at 891.002810s 2024-03-29T11:11:04.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.733+0000 7efe4e7fc640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7efe5013ee90 0x7efe5013b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:57319/0 (socket says 172.21.15.18:57319) 2024-03-29T11:11:04.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.733+0000 7efe4e7fc640 1 -- 172.21.15.18:0/2233321922 learned_addr learned my addr 172.21.15.18:0/2233321922 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:11:04.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.733+0000 7efe4dffb640 1 -- 172.21.15.18:0/2233321922 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2075127025 0 0) 0x7efe500afd10 con 0x7efe5013ee90 2024-03-29T11:11:04.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.733+0000 7efe4dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:11:04.713 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.733+0000 7efe4dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:04.713 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.733+0000 7efe4dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:04.713 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.733+0000 7efe4dffb640 10 monclient(hunting): my global_id is 4293 2024-03-29T11:11:04.713 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.733+0000 7efe4dffb640 1 -- 172.21.15.18:0/2233321922 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe38003520 con 0x7efe5013ee90 2024-03-29T11:11:04.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.735+0000 7efe4dffb640 1 -- 172.21.15.18:0/2233321922 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2975687805 0 0) 0x7efe38003520 con 0x7efe5013ee90 2024-03-29T11:11:04.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.735+0000 7efe4dffb640 1 -- 172.21.15.18:0/2233321922 >> v1:172.21.15.18:6790/0 conn(0x7efe500b4100 legacy=0x7efe500adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:11:04.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.735+0000 7efe4dffb640 1 -- 172.21.15.18:0/2233321922 >> v1:172.21.15.70:6789/0 conn(0x7efe500af900 legacy=0x7efe5014be10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:11:04.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.735+0000 7efe4dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:11:04.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.735+0000 7efe4dffb640 1 monclient: found mon.noname-a 2024-03-29T11:11:04.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.735+0000 7efe4dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:04.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.735+0000 7efe4dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-29T11:11:04.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.735+0000 7efe4dffb640 1 -- 172.21.15.18:0/2233321922 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe500aaf60 con 0x7efe5013ee90 2024-03-29T11:11:04.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.736+0000 7efe4dffb640 10 monclient: _finish_auth 0 2024-03-29T11:11:04.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.736+0000 7efe4dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:11:04.717 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.736+0000 7efe4dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:04.717 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.736+0000 7efe4dffb640 1 -- 172.21.15.18:0/2233321922 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7efe44001690 con 0x7efe5013ee90 2024-03-29T11:11:04.717 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.736+0000 7efe4dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:04.717 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.736+0000 7efe4dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:11:04.719 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.736+0000 7efe4dffb640 10 monclient: dump: 2024-03-29T11:11:04.719 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:11:04.719 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:04.719 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:04.719 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:04.719 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:04.719 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:04.719 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:11:04.719 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:11:04.719 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:11:04.719 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:04.720 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.736+0000 7efe556e4640 5 monclient: authenticate success, global_id 4293 2024-03-29T11:11:04.720 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.736+0000 7efe556e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:04.720 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.737+0000 7efe4dffb640 1 -- 172.21.15.18:0/2233321922 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe44001860 con 0x7efe5013ee90 2024-03-29T11:11:04.720 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.737+0000 7efe4dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:04.720 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.737+0000 7efe4dffb640 1 -- 172.21.15.18:0/2233321922 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7efe44001cb0 con 0x7efe5013ee90 2024-03-29T11:11:04.720 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.737+0000 7efe4dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:04.720 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.737+0000 7efe4dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:11:04.720 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.737+0000 7efe4dffb640 10 monclient: dump: 2024-03-29T11:11:04.720 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:11:04.720 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:04.720 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:04.720 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:04.720 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:04.720 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:04.721 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:11:04.721 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:11:04.721 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:11:04.721 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:04.721 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.737+0000 7efe556e4640 10 monclient: get_monmap_and_config success 2024-03-29T11:11:04.721 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.737+0000 7efe556e4640 4 set_mon_vals no callback set 2024-03-29T11:11:04.721 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.738+0000 7efe556e4640 10 monclient: shutdown 2024-03-29T11:11:04.721 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.738+0000 7efe556e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:04.721 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.738+0000 7efe556e4640 1 -- 172.21.15.18:0/2233321922 >> v1:172.21.15.18:6789/0 conn(0x7efe5013ee90 legacy=0x7efe5013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:04.721 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.739+0000 7efe556e4640 1 -- 172.21.15.18:0/2233321922 shutdown_connections 2024-03-29T11:11:04.721 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.739+0000 7efe556e4640 1 -- 172.21.15.18:0/2233321922 >> 172.21.15.18:0/2233321922 conn(0x7efe500b35e0 msgr2=0x7efe500b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:04.722 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.742+0000 7efe556e4640 1 -- 172.21.15.18:0/2233321922 shutdown_connections 2024-03-29T11:11:04.722 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.743+0000 7efe556e4640 1 -- 172.21.15.18:0/2233321922 wait complete. 2024-03-29T11:11:04.727 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.747+0000 7efe556e4640 10 monclient: build_initial_monmap 2024-03-29T11:11:04.727 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.748+0000 7efe556e4640 10 monclient: monmap: 2024-03-29T11:11:04.727 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:11:04.727 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:11:04.727 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:11:04.727 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:11:04.727 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:04.727 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:04.727 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:11:04.727 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:11:04.727 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:11:04.728 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:04.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.748+0000 7efe556e4640 1 librados: starting msgr at 2024-03-29T11:11:04.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.748+0000 7efe556e4640 1 librados: starting objecter 2024-03-29T11:11:04.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.749+0000 7efe556e4640 1 Processor -- start 2024-03-29T11:11:04.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.749+0000 7efe556e4640 1 -- start start 2024-03-29T11:11:04.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.749+0000 7efe556e4640 1 librados: setting wanted keys 2024-03-29T11:11:04.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.749+0000 7efe556e4640 1 librados: calling monclient init 2024-03-29T11:11:04.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.749+0000 7efe556e4640 10 monclient: init 2024-03-29T11:11:04.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.750+0000 7efe556e4640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:04.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.750+0000 7efe556e4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:11:04.730 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.750+0000 7efe556e4640 10 monclient(hunting): picked mon.noname-c con 0x7efe500af900 addr v1:172.21.15.18:6790/0 2024-03-29T11:11:04.730 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.750+0000 7efe556e4640 10 monclient(hunting): picked mon.noname-b con 0x7efe500b4100 addr v1:172.21.15.70:6789/0 2024-03-29T11:11:04.730 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.750+0000 7efe556e4640 10 monclient(hunting): picked mon.noname-a con 0x7efe5013ee90 addr v1:172.21.15.18:6789/0 2024-03-29T11:11:04.730 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.750+0000 7efe556e4640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe50070d40 con 0x7efe5013ee90 2024-03-29T11:11:04.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.750+0000 7efe556e4640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe5013dd10 con 0x7efe500b4100 2024-03-29T11:11:04.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.750+0000 7efe556e4640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe50070fb0 con 0x7efe500af900 2024-03-29T11:11:04.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.750+0000 7efe556e4640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:04.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.750+0000 7efe556e4640 10 monclient(hunting): authenticate will time out at 891.023787s 2024-03-29T11:11:04.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.753+0000 7efe4effd640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7efe500af900 0x7efe5006eea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:52501/0 (socket says 172.21.15.18:52501) 2024-03-29T11:11:04.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.753+0000 7efe4effd640 1 -- 172.21.15.18:0/3993388350 learned_addr learned my addr 172.21.15.18:0/3993388350 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:11:04.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.754+0000 7efe377fe640 10 client.?.objecter ms_handle_connect 0x7efe500af900 2024-03-29T11:11:04.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.754+0000 7efe377fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:04.734 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.755+0000 7efe377fe640 1 -- 172.21.15.18:0/3993388350 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4175003426 0 0) 0x7efe50070fb0 con 0x7efe500af900 2024-03-29T11:11:04.734 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.755+0000 7efe377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:11:04.734 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.755+0000 7efe377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:04.734 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.755+0000 7efe377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:04.734 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.755+0000 7efe377fe640 10 monclient(hunting): my global_id is 4310 2024-03-29T11:11:04.734 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.755+0000 7efe377fe640 1 -- 172.21.15.18:0/3993388350 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe3c004eb0 con 0x7efe500af900 2024-03-29T11:11:04.735 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.755+0000 7efe377fe640 1 -- 172.21.15.18:0/3993388350 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3478097628 0 0) 0x7efe3c004eb0 con 0x7efe500af900 2024-03-29T11:11:04.735 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.755+0000 7efe377fe640 1 -- 172.21.15.18:0/3993388350 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe50070fb0 con 0x7efe500af900 2024-03-29T11:11:04.735 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.755+0000 7efe377fe640 1 -- 172.21.15.18:0/3993388350 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7efe48004400 con 0x7efe500af900 2024-03-29T11:11:04.735 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.755+0000 7efe377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:04.735 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.755+0000 7efe377fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:11:04.735 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.755+0000 7efe377fe640 10 monclient(hunting): dump: 2024-03-29T11:11:04.735 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:11:04.735 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:04.735 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:04.735 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:04.735 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:04.736 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:04.736 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:11:04.736 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:11:04.736 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:11:04.736 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:04.736 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.755+0000 7efe377fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:11:04.736 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.756+0000 7efe377fe640 10 client.?.objecter ms_handle_connect 0x7efe5013ee90 2024-03-29T11:11:04.736 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.756+0000 7efe377fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:04.736 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.756+0000 7efe377fe640 1 -- 172.21.15.18:0/3993388350 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2139976432 0 0) 0x7efe50070fb0 con 0x7efe500af900 2024-03-29T11:11:04.736 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.756+0000 7efe377fe640 1 -- 172.21.15.18:0/3993388350 >> v1:172.21.15.70:6789/0 conn(0x7efe500b4100 legacy=0x7efe50071990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:11:04.739 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.759+0000 7efe377fe640 1 -- 172.21.15.18:0/3993388350 >> v1:172.21.15.18:6789/0 conn(0x7efe5013ee90 legacy=0x7efe501e26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:04.739 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.759+0000 7efe377fe640 10 monclient: _finish_hunting 0 2024-03-29T11:11:04.739 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.759+0000 7efe377fe640 1 monclient: found mon.c 2024-03-29T11:11:04.739 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.759+0000 7efe377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:04.739 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.759+0000 7efe377fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:11:04.739 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.759+0000 7efe377fe640 1 -- 172.21.15.18:0/3993388350 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe50071290 con 0x7efe500af900 2024-03-29T11:11:04.739 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.759+0000 7efe377fe640 10 monclient: _finish_auth 0 2024-03-29T11:11:04.739 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.759+0000 7efe377fe640 10 monclient: _check_auth_tickets 2024-03-29T11:11:04.739 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.759+0000 7efe377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:04.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.759+0000 7efe556e4640 5 monclient: authenticate success, global_id 4310 2024-03-29T11:11:04.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.759+0000 7efe556e4640 10 monclient: _renew_subs 2024-03-29T11:11:04.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.759+0000 7efe556e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:11:04.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.759+0000 7efe556e4640 1 -- 172.21.15.18:0/3993388350 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe50071570 con 0x7efe500af900 2024-03-29T11:11:04.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.759+0000 7efe556e4640 10 client.4310.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:11:04.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.759+0000 7efe556e4640 10 monclient: _renew_subs 2024-03-29T11:11:04.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.759+0000 7efe556e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:11:04.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.759+0000 7efe556e4640 1 -- 172.21.15.18:0/3993388350 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efe5012bd30 con 0x7efe500af900 2024-03-29T11:11:04.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.759+0000 7efe556e4640 1 librados: init done 2024-03-29T11:11:04.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.759+0000 7efe377fe640 1 -- 172.21.15.18:0/3993388350 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe48004660 con 0x7efe500af900 2024-03-29T11:11:04.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.759+0000 7efe377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:04.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.759+0000 7efe377fe640 1 -- 172.21.15.18:0/3993388350 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7efe480016a0 con 0x7efe500af900 2024-03-29T11:11:04.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.759+0000 7efe377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:04.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.759+0000 7efe377fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:04.741 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.759+0000 7efe377fe640 10 monclient: dump: 2024-03-29T11:11:04.741 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:11:04.741 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:04.741 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:04.741 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:04.741 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:04.741 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:04.741 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:11:04.741 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:11:04.741 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:11:04.741 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:04.741 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.760+0000 7efe4cff9640 4 set_mon_vals no callback set 2024-03-29T11:11:04.742 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.761+0000 7efe556e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:04.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.761+0000 7efe556e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:04.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.761+0000 7efe556e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:11:04.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.761+0000 7efe556e4640 1 -- 172.21.15.18:0/3993388350 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe30005080 con 0x7efe500af900 2024-03-29T11:11:04.744 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.762+0000 7efe377fe640 1 -- 172.21.15.18:0/3993388350 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7efe48001800 con 0x7efe500af900 2024-03-29T11:11:04.744 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.762+0000 7efe377fe640 1 -- 172.21.15.18:0/3993388350 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7efe48084f00 con 0x7efe500af900 2024-03-29T11:11:04.744 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.762+0000 7efe377fe640 10 client.4310.objecter ms_dispatch 0x7efe50000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:11:04.744 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.762+0000 7efe377fe640 3 client.4310.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:11:04.744 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.762+0000 7efe377fe640 3 client.4310.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:11:04.745 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.763+0000 7efe377fe640 20 client.4310.objecter dump_active .. 0 homeless 2024-03-29T11:11:04.745 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.765+0000 7efe4e7fc640 10 monclient: get_auth_request con 0x7efe3c06db60 auth_method 0 2024-03-29T11:11:04.746 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.766+0000 7efe377fe640 10 client.4310.objecter ms_handle_connect 0x7efe3c06db60 2024-03-29T11:11:04.749 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.770+0000 7efe377fe640 1 -- 172.21.15.18:0/3993388350 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efe480507e0 con 0x7efe500af900 2024-03-29T11:11:04.749 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.770+0000 7efe377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:04.749 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:04.770+0000 7efe377fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:11:05.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:05.112+0000 7efe556e4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:05.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:05.112+0000 7efe556e4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:05.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:05.112+0000 7efe556e4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:11:05.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:05.112+0000 7efe556e4640 1 -- 172.21.15.18:0/3993388350 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe30005330 con 0x7efe500af900 2024-03-29T11:11:05.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:05.115+0000 7efe377fe640 1 -- 172.21.15.18:0/3993388350 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7efe48052710 con 0x7efe500af900 2024-03-29T11:11:05.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:05.115+0000 7efe377fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:05.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:05.115+0000 7efe377fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:11:05.094 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:11:05.094 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:11:05.100 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:05.118+0000 7efe556e4640 10 librados: watch_flush enter 2024-03-29T11:11:05.100 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:05.120+0000 7efe556e4640 10 librados: watch_flush exit 2024-03-29T11:11:05.100 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:05.120+0000 7efe556e4640 20 client.4310.objecter shutdown clearing up homeless session... 2024-03-29T11:11:05.100 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:05.120+0000 7efe556e4640 10 client.4310.objecter successfully canceled tick 2024-03-29T11:11:05.101 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:05.122+0000 7efe556e4640 1 -- 172.21.15.18:0/3993388350 >> v1:172.21.15.70:6816/3101871562 conn(0x7efe3c06db60 legacy=0x7efe3c070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:05.101 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:05.122+0000 7efe556e4640 10 monclient: shutdown 2024-03-29T11:11:05.101 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:05.122+0000 7efe556e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:05.101 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:05.122+0000 7efe556e4640 1 -- 172.21.15.18:0/3993388350 >> v1:172.21.15.18:6790/0 conn(0x7efe500af900 legacy=0x7efe5006eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:05.101 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:05.122+0000 7efe556e4640 1 -- 172.21.15.18:0/3993388350 shutdown_connections 2024-03-29T11:11:05.101 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:05.122+0000 7efe556e4640 1 -- 172.21.15.18:0/3993388350 >> 172.21.15.18:0/3993388350 conn(0x7efe500b35e0 msgr2=0x7efe500acf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:05.102 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:05.123+0000 7efe556e4640 1 -- 172.21.15.18:0/3993388350 shutdown_connections 2024-03-29T11:11:05.102 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:05.123+0000 7efe556e4640 1 -- 172.21.15.18:0/3993388350 wait complete. 2024-03-29T11:11:05.106 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:05.123+0000 7efe556e4640 1 librados: shutdown 2024-03-29T11:11:05.122 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:11:11.930 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:11:12.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.100+0000 7fa3175c4640 10 monclient: get_monmap_and_config 2024-03-29T11:11:12.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.100+0000 7fa3175c4640 10 monclient: build_initial_monmap 2024-03-29T11:11:12.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.100+0000 7fa3175c4640 10 monclient: monmap: 2024-03-29T11:11:12.082 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:11:12.082 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:12.082 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:11:12.101177+0000 2024-03-29T11:11:12.082 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:11:12.101177+0000 2024-03-29T11:11:12.082 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:12.082 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:12.082 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:11:12.082 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:11:12.082 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:11:12.082 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:12.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.102+0000 7fa3175c4640 1 Processor -- start 2024-03-29T11:11:12.086 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.105+0000 7fa3175c4640 1 -- start start 2024-03-29T11:11:12.086 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.105+0000 7fa3175c4640 10 monclient: init 2024-03-29T11:11:12.086 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.106+0000 7fa3175c4640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:12.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.106+0000 7fa3175c4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:11:12.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.106+0000 7fa3175c4640 10 monclient(hunting): picked mon.noname-b con 0x7fa3100a9ac0 addr v1:172.21.15.18:6790/0 2024-03-29T11:11:12.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.106+0000 7fa3175c4640 10 monclient(hunting): picked mon.noname-a con 0x7fa31013ee90 addr v1:172.21.15.18:6789/0 2024-03-29T11:11:12.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.106+0000 7fa3175c4640 10 monclient(hunting): picked mon.noname-c con 0x7fa3100af900 addr v1:172.21.15.70:6789/0 2024-03-29T11:11:12.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.106+0000 7fa3175c4640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3100afd10 con 0x7fa31013ee90 2024-03-29T11:11:12.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.106+0000 7fa3175c4640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3100af6f0 con 0x7fa3100af900 2024-03-29T11:11:12.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.106+0000 7fa3175c4640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3100aac80 con 0x7fa3100a9ac0 2024-03-29T11:11:12.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.106+0000 7fa3175c4640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:12.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.106+0000 7fa3175c4640 10 monclient(hunting): authenticate will time out at 898.378159s 2024-03-29T11:11:12.088 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.109+0000 7fa314b5f640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fa31013ee90 0x7fa31013b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:55297/0 (socket says 172.21.15.18:55297) 2024-03-29T11:11:12.088 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.109+0000 7fa314b5f640 1 -- 172.21.15.18:0/58597315 learned_addr learned my addr 172.21.15.18:0/58597315 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:11:12.091 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.111+0000 7fa307fff640 1 -- 172.21.15.18:0/58597315 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2878915853 0 0) 0x7fa3100aac80 con 0x7fa3100a9ac0 2024-03-29T11:11:12.091 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.112+0000 7fa307fff640 10 monclient(hunting): _negotiate 2024-03-29T11:11:12.091 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.112+0000 7fa307fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:12.091 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.112+0000 7fa307fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:12.091 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.112+0000 7fa307fff640 10 monclient(hunting): my global_id is 4313 2024-03-29T11:11:12.091 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.112+0000 7fa307fff640 1 -- 172.21.15.18:0/58597315 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2fc003520 con 0x7fa3100a9ac0 2024-03-29T11:11:12.091 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.112+0000 7fa307fff640 1 -- 172.21.15.18:0/58597315 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2314316410 0 0) 0x7fa2fc003520 con 0x7fa3100a9ac0 2024-03-29T11:11:12.091 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.112+0000 7fa307fff640 1 -- 172.21.15.18:0/58597315 >> v1:172.21.15.70:6789/0 conn(0x7fa3100af900 legacy=0x7fa310149b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:11:12.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.114+0000 7fa307fff640 1 -- 172.21.15.18:0/58597315 >> v1:172.21.15.18:6789/0 conn(0x7fa31013ee90 legacy=0x7fa31013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:12.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.114+0000 7fa307fff640 10 monclient: _finish_hunting 0 2024-03-29T11:11:12.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.114+0000 7fa307fff640 1 monclient: found mon.noname-b 2024-03-29T11:11:12.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.114+0000 7fa307fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:12.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.114+0000 7fa307fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:11:12.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.114+0000 7fa307fff640 1 -- 172.21.15.18:0/58597315 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3100aaf60 con 0x7fa3100a9ac0 2024-03-29T11:11:12.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.115+0000 7fa307fff640 10 monclient: _finish_auth 0 2024-03-29T11:11:12.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.115+0000 7fa307fff640 10 monclient: _check_auth_tickets 2024-03-29T11:11:12.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.115+0000 7fa307fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:12.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.115+0000 7fa3175c4640 5 monclient: authenticate success, global_id 4313 2024-03-29T11:11:12.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.115+0000 7fa3175c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:12.095 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.115+0000 7fa307fff640 1 -- 172.21.15.18:0/58597315 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fa30c001690 con 0x7fa3100a9ac0 2024-03-29T11:11:12.095 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.115+0000 7fa307fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:12.095 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.115+0000 7fa307fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:11:12.095 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.115+0000 7fa307fff640 10 monclient: dump: 2024-03-29T11:11:12.095 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:11:12.095 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:12.095 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:12.095 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:12.095 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:12.095 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:12.095 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:11:12.095 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:11:12.095 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:11:12.095 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:12.095 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.115+0000 7fa3175c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:12.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.115+0000 7fa307fff640 1 -- 172.21.15.18:0/58597315 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa30c001920 con 0x7fa3100a9ac0 2024-03-29T11:11:12.096 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.115+0000 7fa307fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:12.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.115+0000 7fa3175c4640 10 monclient: get_monmap_and_config success 2024-03-29T11:11:12.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.115+0000 7fa3175c4640 4 set_mon_vals no callback set 2024-03-29T11:11:12.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.116+0000 7fa307fff640 1 -- 172.21.15.18:0/58597315 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fa30c001cf0 con 0x7fa3100a9ac0 2024-03-29T11:11:12.098 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.116+0000 7fa307fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:12.098 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.116+0000 7fa307fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:12.098 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.116+0000 7fa307fff640 10 monclient: dump: 2024-03-29T11:11:12.098 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:11:12.098 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:12.098 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:12.098 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:12.098 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:12.098 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:12.099 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:11:12.099 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:11:12.099 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:11:12.099 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:12.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.116+0000 7fa3175c4640 10 monclient: shutdown 2024-03-29T11:11:12.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.116+0000 7fa3175c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:12.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.116+0000 7fa3175c4640 1 -- 172.21.15.18:0/58597315 >> v1:172.21.15.18:6790/0 conn(0x7fa3100a9ac0 legacy=0x7fa3100adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:12.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.118+0000 7fa3175c4640 1 -- 172.21.15.18:0/58597315 shutdown_connections 2024-03-29T11:11:12.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.118+0000 7fa3175c4640 1 -- 172.21.15.18:0/58597315 >> 172.21.15.18:0/58597315 conn(0x7fa3100b35e0 msgr2=0x7fa3100b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:12.100 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.120+0000 7fa3175c4640 1 -- 172.21.15.18:0/58597315 shutdown_connections 2024-03-29T11:11:12.102 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.121+0000 7fa3175c4640 1 -- 172.21.15.18:0/58597315 wait complete. 2024-03-29T11:11:12.102 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.121+0000 7fa3175c4640 10 monclient: build_initial_monmap 2024-03-29T11:11:12.102 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.121+0000 7fa3175c4640 10 monclient: monmap: 2024-03-29T11:11:12.103 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:11:12.103 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:11:12.103 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:11:12.103 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:11:12.103 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:12.103 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:12.103 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:11:12.103 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:11:12.103 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:11:12.103 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:12.103 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.122+0000 7fa3175c4640 1 librados: starting msgr at 2024-03-29T11:11:12.103 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.122+0000 7fa3175c4640 1 librados: starting objecter 2024-03-29T11:11:12.103 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.122+0000 7fa3175c4640 1 Processor -- start 2024-03-29T11:11:12.103 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.122+0000 7fa3175c4640 1 -- start start 2024-03-29T11:11:12.103 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.122+0000 7fa3175c4640 1 librados: setting wanted keys 2024-03-29T11:11:12.104 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.122+0000 7fa3175c4640 1 librados: calling monclient init 2024-03-29T11:11:12.104 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.122+0000 7fa3175c4640 10 monclient: init 2024-03-29T11:11:12.104 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.123+0000 7fa3175c4640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:12.104 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.123+0000 7fa3175c4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:11:12.104 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.123+0000 7fa3175c4640 10 monclient(hunting): picked mon.noname-b con 0x7fa3100a9ac0 addr v1:172.21.15.70:6789/0 2024-03-29T11:11:12.104 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.123+0000 7fa3175c4640 10 monclient(hunting): picked mon.noname-c con 0x7fa3100af900 addr v1:172.21.15.18:6790/0 2024-03-29T11:11:12.104 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.123+0000 7fa3175c4640 10 monclient(hunting): picked mon.noname-a con 0x7fa31013ee90 addr v1:172.21.15.18:6789/0 2024-03-29T11:11:12.104 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.123+0000 7fa3175c4640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa310141b30 con 0x7fa31013ee90 2024-03-29T11:11:12.104 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.123+0000 7fa3175c4640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa31013d680 con 0x7fa3100a9ac0 2024-03-29T11:11:12.104 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.123+0000 7fa3175c4640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa31012f900 con 0x7fa3100af900 2024-03-29T11:11:12.104 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.123+0000 7fa3175c4640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:12.104 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.123+0000 7fa3175c4640 10 monclient(hunting): authenticate will time out at 898.395422s 2024-03-29T11:11:12.105 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.127+0000 7fa315b61640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fa31013ee90 0x7fa3101e6640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:56797/0 (socket says 172.21.15.18:56797) 2024-03-29T11:11:12.105 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.127+0000 7fa315b61640 1 -- 172.21.15.18:0/1125894834 learned_addr learned my addr 172.21.15.18:0/1125894834 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:11:12.109 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.131+0000 7fa305ffb640 10 client.?.objecter ms_handle_connect 0x7fa3100af900 2024-03-29T11:11:12.109 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.131+0000 7fa305ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:12.109 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.131+0000 7fa305ffb640 10 client.?.objecter ms_handle_connect 0x7fa3100a9ac0 2024-03-29T11:11:12.109 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.131+0000 7fa305ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:12.111 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.132+0000 7fa305ffb640 10 client.?.objecter ms_handle_connect 0x7fa31013ee90 2024-03-29T11:11:12.111 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.132+0000 7fa305ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:12.111 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.132+0000 7fa305ffb640 1 -- 172.21.15.18:0/1125894834 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2806838146 0 0) 0x7fa31012f900 con 0x7fa3100af900 2024-03-29T11:11:12.111 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.133+0000 7fa305ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:11:12.111 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.133+0000 7fa305ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:12.111 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.133+0000 7fa305ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:12.111 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.133+0000 7fa305ffb640 10 monclient(hunting): my global_id is 4316 2024-03-29T11:11:12.112 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.133+0000 7fa305ffb640 1 -- 172.21.15.18:0/1125894834 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2f4004eb0 con 0x7fa3100af900 2024-03-29T11:11:12.112 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.133+0000 7fa305ffb640 1 -- 172.21.15.18:0/1125894834 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2361643843 0 0) 0x7fa310141b30 con 0x7fa31013ee90 2024-03-29T11:11:12.112 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.133+0000 7fa305ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:11:12.112 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.133+0000 7fa305ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:12.112 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.133+0000 7fa305ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:12.112 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.133+0000 7fa305ffb640 10 monclient(hunting): my global_id is 4302 2024-03-29T11:11:12.112 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.133+0000 7fa305ffb640 1 -- 172.21.15.18:0/1125894834 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa31012f900 con 0x7fa31013ee90 2024-03-29T11:11:12.114 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.135+0000 7fa305ffb640 1 -- 172.21.15.18:0/1125894834 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1033419609 0 0) 0x7fa2f4004eb0 con 0x7fa3100af900 2024-03-29T11:11:12.114 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.135+0000 7fa305ffb640 1 -- 172.21.15.18:0/1125894834 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa310141b30 con 0x7fa3100af900 2024-03-29T11:11:12.114 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.135+0000 7fa305ffb640 1 -- 172.21.15.18:0/1125894834 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fa3000017e0 con 0x7fa3100af900 2024-03-29T11:11:12.114 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.135+0000 7fa305ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:12.114 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.135+0000 7fa305ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:11:12.114 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.135+0000 7fa305ffb640 10 monclient(hunting): dump: 2024-03-29T11:11:12.114 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:11:12.114 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:12.114 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:12.114 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:12.115 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:12.115 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:12.115 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:11:12.115 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:11:12.115 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:11:12.115 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:12.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.135+0000 7fa305ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:11:12.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.135+0000 7fa305ffb640 1 -- 172.21.15.18:0/1125894834 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1006593008 0 0) 0x7fa31013d680 con 0x7fa3100a9ac0 2024-03-29T11:11:12.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.136+0000 7fa305ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:11:12.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.136+0000 7fa305ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:12.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.136+0000 7fa305ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:12.115 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.136+0000 7fa305ffb640 10 monclient(hunting): my global_id is 4303 2024-03-29T11:11:12.117 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.136+0000 7fa305ffb640 1 -- 172.21.15.18:0/1125894834 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2f4004eb0 con 0x7fa3100a9ac0 2024-03-29T11:11:12.117 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.136+0000 7fa305ffb640 1 -- 172.21.15.18:0/1125894834 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 949008039 0 0) 0x7fa31012f900 con 0x7fa31013ee90 2024-03-29T11:11:12.117 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.136+0000 7fa305ffb640 1 -- 172.21.15.18:0/1125894834 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa31013d680 con 0x7fa31013ee90 2024-03-29T11:11:12.117 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.136+0000 7fa305ffb640 1 -- 172.21.15.18:0/1125894834 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fa308001af0 con 0x7fa31013ee90 2024-03-29T11:11:12.117 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.136+0000 7fa305ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:12.117 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.136+0000 7fa305ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:11:12.117 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.136+0000 7fa305ffb640 10 monclient(hunting): dump: 2024-03-29T11:11:12.117 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:11:12.117 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:12.117 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:12.118 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:12.118 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:12.118 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:12.118 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:11:12.118 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:11:12.118 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:11:12.118 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:12.118 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.136+0000 7fa305ffb640 1 -- 172.21.15.18:0/1125894834 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2654837542 0 0) 0x7fa310141b30 con 0x7fa3100af900 2024-03-29T11:11:12.118 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.136+0000 7fa305ffb640 1 -- 172.21.15.18:0/1125894834 >> v1:172.21.15.70:6789/0 conn(0x7fa3100a9ac0 legacy=0x7fa31012c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:12.118 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.136+0000 7fa305ffb640 1 -- 172.21.15.18:0/1125894834 >> v1:172.21.15.18:6789/0 conn(0x7fa31013ee90 legacy=0x7fa3101e6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:12.118 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.136+0000 7fa305ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:11:12.118 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.136+0000 7fa305ffb640 1 monclient: found mon.c 2024-03-29T11:11:12.119 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.136+0000 7fa305ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:12.119 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.136+0000 7fa305ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:11:12.119 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.136+0000 7fa305ffb640 1 -- 172.21.15.18:0/1125894834 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa310141d60 con 0x7fa3100af900 2024-03-29T11:11:12.119 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.136+0000 7fa305ffb640 10 monclient: _finish_auth 0 2024-03-29T11:11:12.119 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.136+0000 7fa305ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:11:12.119 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.136+0000 7fa305ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:12.119 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.136+0000 7fa3175c4640 5 monclient: authenticate success, global_id 4316 2024-03-29T11:11:12.119 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.136+0000 7fa3175c4640 10 monclient: _renew_subs 2024-03-29T11:11:12.119 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.136+0000 7fa3175c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:11:12.119 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.136+0000 7fa3175c4640 1 -- 172.21.15.18:0/1125894834 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa310142040 con 0x7fa3100af900 2024-03-29T11:11:12.119 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.136+0000 7fa3175c4640 10 client.4316.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:11:12.119 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.136+0000 7fa3175c4640 10 monclient: _renew_subs 2024-03-29T11:11:12.119 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.137+0000 7fa3175c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:11:12.119 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.137+0000 7fa3175c4640 1 -- 172.21.15.18:0/1125894834 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3101411c0 con 0x7fa3100af900 2024-03-29T11:11:12.119 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.137+0000 7fa3175c4640 1 librados: init done 2024-03-29T11:11:12.120 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.137+0000 7fa305ffb640 1 -- 172.21.15.18:0/1125894834 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa300001d10 con 0x7fa3100af900 2024-03-29T11:11:12.120 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.137+0000 7fa305ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:12.120 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.137+0000 7fa305ffb640 1 -- 172.21.15.18:0/1125894834 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fa300004980 con 0x7fa3100af900 2024-03-29T11:11:12.120 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.137+0000 7fa305ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:12.120 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.137+0000 7fa305ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:12.120 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.137+0000 7fa305ffb640 10 monclient: dump: 2024-03-29T11:11:12.120 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:11:12.120 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:12.120 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:12.120 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:12.120 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:12.120 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:12.120 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:11:12.120 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:11:12.121 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:11:12.121 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:12.121 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.137+0000 7fa306ffd640 4 set_mon_vals no callback set 2024-03-29T11:11:12.121 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.139+0000 7fa305ffb640 1 -- 172.21.15.18:0/1125894834 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7fa30001a950 con 0x7fa3100af900 2024-03-29T11:11:12.122 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.140+0000 7fa3175c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:12.122 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.140+0000 7fa3175c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:12.122 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.140+0000 7fa3175c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:11:12.122 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.140+0000 7fa3175c4640 1 -- 172.21.15.18:0/1125894834 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2f0005080 con 0x7fa3100af900 2024-03-29T11:11:12.122 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.140+0000 7fa305ffb640 1 -- 172.21.15.18:0/1125894834 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7fa300084960 con 0x7fa3100af900 2024-03-29T11:11:12.122 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.140+0000 7fa305ffb640 10 client.4316.objecter ms_dispatch 0x7fa310000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:11:12.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.140+0000 7fa305ffb640 3 client.4316.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:11:12.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.140+0000 7fa305ffb640 3 client.4316.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:11:12.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.140+0000 7fa305ffb640 20 client.4316.objecter dump_active .. 0 homeless 2024-03-29T11:11:12.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.141+0000 7fa315360640 10 monclient: get_auth_request con 0x7fa2f406e000 auth_method 0 2024-03-29T11:11:12.123 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.144+0000 7fa305ffb640 10 client.4316.objecter ms_handle_connect 0x7fa2f406e000 2024-03-29T11:11:12.129 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.151+0000 7fa305ffb640 1 -- 172.21.15.18:0/1125894834 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa300050070 con 0x7fa3100af900 2024-03-29T11:11:12.129 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.151+0000 7fa305ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:12.129 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.151+0000 7fa305ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:11:12.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.466+0000 7fa3175c4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:12.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.466+0000 7fa3175c4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:12.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.466+0000 7fa3175c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:11:12.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.466+0000 7fa3175c4640 1 -- 172.21.15.18:0/1125894834 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2f0005330 con 0x7fa3100af900 2024-03-29T11:11:12.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.468+0000 7fa305ffb640 1 -- 172.21.15.18:0/1125894834 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7fa300052170 con 0x7fa3100af900 2024-03-29T11:11:12.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.468+0000 7fa305ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:12.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.468+0000 7fa305ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:11:12.446 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:11:12.446 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:11:12.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.471+0000 7fa3175c4640 10 librados: watch_flush enter 2024-03-29T11:11:12.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.471+0000 7fa3175c4640 10 librados: watch_flush exit 2024-03-29T11:11:12.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.471+0000 7fa3175c4640 20 client.4316.objecter shutdown clearing up homeless session... 2024-03-29T11:11:12.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.471+0000 7fa3175c4640 10 client.4316.objecter successfully canceled tick 2024-03-29T11:11:12.450 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.471+0000 7fa3175c4640 1 -- 172.21.15.18:0/1125894834 >> v1:172.21.15.70:6816/3101871562 conn(0x7fa2f406e000 legacy=0x7fa2f40704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:12.450 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.471+0000 7fa3175c4640 10 monclient: shutdown 2024-03-29T11:11:12.450 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.471+0000 7fa3175c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:12.450 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.471+0000 7fa3175c4640 1 -- 172.21.15.18:0/1125894834 >> v1:172.21.15.18:6790/0 conn(0x7fa3100af900 legacy=0x7fa31012cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:12.450 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.473+0000 7fa3175c4640 1 -- 172.21.15.18:0/1125894834 shutdown_connections 2024-03-29T11:11:12.450 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.473+0000 7fa3175c4640 1 -- 172.21.15.18:0/1125894834 >> 172.21.15.18:0/1125894834 conn(0x7fa3100b35e0 msgr2=0x7fa3100acff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:12.451 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.473+0000 7fa3175c4640 1 -- 172.21.15.18:0/1125894834 shutdown_connections 2024-03-29T11:11:12.452 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.473+0000 7fa3175c4640 1 -- 172.21.15.18:0/1125894834 wait complete. 2024-03-29T11:11:12.452 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:12.474+0000 7fa3175c4640 1 librados: shutdown 2024-03-29T11:11:12.470 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:11:19.277 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:11:19.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.451+0000 7f2d00015640 10 monclient: get_monmap_and_config 2024-03-29T11:11:19.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.451+0000 7f2d00015640 10 monclient: build_initial_monmap 2024-03-29T11:11:19.430 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.451+0000 7f2d00015640 10 monclient: monmap: 2024-03-29T11:11:19.430 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:11:19.430 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:19.430 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:11:19.452473+0000 2024-03-29T11:11:19.430 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:11:19.452473+0000 2024-03-29T11:11:19.430 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:19.430 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:19.430 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:11:19.430 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:11:19.430 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:11:19.430 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:19.431 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.452+0000 7f2d00015640 1 Processor -- start 2024-03-29T11:11:19.431 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.452+0000 7f2d00015640 1 -- start start 2024-03-29T11:11:19.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.454+0000 7f2d00015640 10 monclient: init 2024-03-29T11:11:19.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.454+0000 7f2d00015640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:19.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.455+0000 7f2d00015640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:11:19.437 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.457+0000 7f2d00015640 10 monclient(hunting): picked mon.noname-a con 0x7f2cf80b0860 addr v1:172.21.15.18:6789/0 2024-03-29T11:11:19.437 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.457+0000 7f2d00015640 10 monclient(hunting): picked mon.noname-c con 0x7f2cf813ef30 addr v1:172.21.15.70:6789/0 2024-03-29T11:11:19.437 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.457+0000 7f2d00015640 10 monclient(hunting): picked mon.noname-b con 0x7f2cf80af8e0 addr v1:172.21.15.18:6790/0 2024-03-29T11:11:19.437 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.457+0000 7f2d00015640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cf80b0c40 con 0x7f2cf80b0860 2024-03-29T11:11:19.437 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.457+0000 7f2d00015640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cf80aed10 con 0x7f2cf813ef30 2024-03-29T11:11:19.437 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.457+0000 7f2d00015640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cf80afd50 con 0x7f2cf80af8e0 2024-03-29T11:11:19.437 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.457+0000 7f2d00015640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:19.437 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.457+0000 7f2d00015640 10 monclient(hunting): authenticate will time out at 905.729017s 2024-03-29T11:11:19.437 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.458+0000 7f2cfe5b2640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f2cf80af8e0 0x7f2cf8149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:40584/0 (socket says 172.21.15.18:40584) 2024-03-29T11:11:19.437 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.458+0000 7f2cfe5b2640 1 -- 172.21.15.18:0/3427914704 learned_addr learned my addr 172.21.15.18:0/3427914704 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:11:19.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.460+0000 7f2cfcdaf640 1 -- 172.21.15.18:0/3427914704 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2519662503 0 0) 0x7f2cf80b0c40 con 0x7f2cf80b0860 2024-03-29T11:11:19.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.460+0000 7f2cfcdaf640 10 monclient(hunting): _negotiate 2024-03-29T11:11:19.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.460+0000 7f2cfcdaf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:19.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.460+0000 7f2cfcdaf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:19.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.460+0000 7f2cfcdaf640 10 monclient(hunting): my global_id is 4305 2024-03-29T11:11:19.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.460+0000 7f2cfcdaf640 1 -- 172.21.15.18:0/3427914704 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ce4003520 con 0x7f2cf80b0860 2024-03-29T11:11:19.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.460+0000 7f2cfcdaf640 1 -- 172.21.15.18:0/3427914704 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4201918514 0 0) 0x7f2cf80aed10 con 0x7f2cf813ef30 2024-03-29T11:11:19.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.460+0000 7f2cfcdaf640 10 monclient(hunting): _negotiate 2024-03-29T11:11:19.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.460+0000 7f2cfcdaf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:19.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.460+0000 7f2cfcdaf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:19.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.460+0000 7f2cfcdaf640 10 monclient(hunting): my global_id is 4306 2024-03-29T11:11:19.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.460+0000 7f2cfcdaf640 1 -- 172.21.15.18:0/3427914704 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cf80b0c40 con 0x7f2cf813ef30 2024-03-29T11:11:19.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.460+0000 7f2cfcdaf640 1 -- 172.21.15.18:0/3427914704 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3899214959 0 0) 0x7f2cf80afd50 con 0x7f2cf80af8e0 2024-03-29T11:11:19.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.461+0000 7f2cfcdaf640 10 monclient(hunting): _negotiate 2024-03-29T11:11:19.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.461+0000 7f2cfcdaf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:19.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.461+0000 7f2cfcdaf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:19.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.461+0000 7f2cfcdaf640 10 monclient(hunting): my global_id is 4319 2024-03-29T11:11:19.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.461+0000 7f2cfcdaf640 1 -- 172.21.15.18:0/3427914704 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cf80aed10 con 0x7f2cf80af8e0 2024-03-29T11:11:19.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.461+0000 7f2cfcdaf640 1 -- 172.21.15.18:0/3427914704 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 935294028 0 0) 0x7f2cf80b0c40 con 0x7f2cf813ef30 2024-03-29T11:11:19.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.461+0000 7f2cfcdaf640 1 -- 172.21.15.18:0/3427914704 >> v1:172.21.15.18:6790/0 conn(0x7f2cf80af8e0 legacy=0x7f2cf8149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:19.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.461+0000 7f2cfcdaf640 1 -- 172.21.15.18:0/3427914704 >> v1:172.21.15.18:6789/0 conn(0x7f2cf80b0860 legacy=0x7f2cf813e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:19.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.461+0000 7f2cfcdaf640 10 monclient: _finish_hunting 0 2024-03-29T11:11:19.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.461+0000 7f2cfcdaf640 1 monclient: found mon.noname-c 2024-03-29T11:11:19.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.461+0000 7f2cfcdaf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:19.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.461+0000 7f2cfcdaf640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.70:6789/0 2024-03-29T11:11:19.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.461+0000 7f2cfcdaf640 1 -- 172.21.15.18:0/3427914704 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf80ab1b0 con 0x7f2cf813ef30 2024-03-29T11:11:19.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.461+0000 7f2cfcdaf640 10 monclient: _finish_auth 0 2024-03-29T11:11:19.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.461+0000 7f2cfcdaf640 10 monclient: _check_auth_tickets 2024-03-29T11:11:19.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.461+0000 7f2cfcdaf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:19.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.461+0000 7f2cfcdaf640 1 -- 172.21.15.18:0/3427914704 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f2ce8001670 con 0x7f2cf813ef30 2024-03-29T11:11:19.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.461+0000 7f2cfcdaf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:19.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.461+0000 7f2cfcdaf640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:11:19.444 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.461+0000 7f2cfcdaf640 10 monclient: dump: 2024-03-29T11:11:19.444 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:11:19.444 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:19.444 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:19.444 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:19.444 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:19.444 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:19.444 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:11:19.444 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:11:19.444 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:11:19.444 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:19.444 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.462+0000 7f2d00015640 5 monclient: authenticate success, global_id 4306 2024-03-29T11:11:19.444 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.462+0000 7f2d00015640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:19.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.464+0000 7f2cfcdaf640 1 -- 172.21.15.18:0/3427914704 <== mon.1 v1:172.21.15.70:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ce80017d0 con 0x7f2cf813ef30 2024-03-29T11:11:19.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.464+0000 7f2cfcdaf640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:19.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.464+0000 7f2cfcdaf640 1 -- 172.21.15.18:0/3427914704 <== mon.1 v1:172.21.15.70:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f2ce8001be0 con 0x7f2cf813ef30 2024-03-29T11:11:19.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.464+0000 7f2cfcdaf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:19.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.464+0000 7f2cfcdaf640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:11:19.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.464+0000 7f2cfcdaf640 10 monclient: dump: 2024-03-29T11:11:19.446 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:11:19.446 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:19.446 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:19.446 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:19.446 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:19.446 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:19.446 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:11:19.446 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:11:19.446 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:11:19.446 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:19.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.464+0000 7f2d00015640 10 monclient: get_monmap_and_config success 2024-03-29T11:11:19.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.464+0000 7f2d00015640 4 set_mon_vals no callback set 2024-03-29T11:11:19.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.466+0000 7f2d00015640 10 monclient: shutdown 2024-03-29T11:11:19.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.466+0000 7f2d00015640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:19.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.466+0000 7f2d00015640 1 -- 172.21.15.18:0/3427914704 >> v1:172.21.15.70:6789/0 conn(0x7f2cf813ef30 legacy=0x7f2cf80af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:19.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.466+0000 7f2d00015640 1 -- 172.21.15.18:0/3427914704 shutdown_connections 2024-03-29T11:11:19.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.466+0000 7f2d00015640 1 -- 172.21.15.18:0/3427914704 >> 172.21.15.18:0/3427914704 conn(0x7f2cf806b660 msgr2=0x7f2cf806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:19.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.470+0000 7f2d00015640 1 -- 172.21.15.18:0/3427914704 shutdown_connections 2024-03-29T11:11:19.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.471+0000 7f2d00015640 1 -- 172.21.15.18:0/3427914704 wait complete. 2024-03-29T11:11:19.450 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.472+0000 7f2d00015640 10 monclient: build_initial_monmap 2024-03-29T11:11:19.450 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.472+0000 7f2d00015640 10 monclient: monmap: 2024-03-29T11:11:19.451 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:11:19.451 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:11:19.451 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:11:19.451 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:11:19.451 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:19.451 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:19.451 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:11:19.451 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:11:19.451 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:11:19.451 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:19.451 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.472+0000 7f2d00015640 1 librados: starting msgr at 2024-03-29T11:11:19.451 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.472+0000 7f2d00015640 1 librados: starting objecter 2024-03-29T11:11:19.453 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.472+0000 7f2d00015640 1 Processor -- start 2024-03-29T11:11:19.453 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.473+0000 7f2d00015640 1 -- start start 2024-03-29T11:11:19.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.473+0000 7f2d00015640 1 librados: setting wanted keys 2024-03-29T11:11:19.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.473+0000 7f2d00015640 1 librados: calling monclient init 2024-03-29T11:11:19.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.473+0000 7f2d00015640 10 monclient: init 2024-03-29T11:11:19.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.474+0000 7f2d00015640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:19.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.474+0000 7f2d00015640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:11:19.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.474+0000 7f2d00015640 10 monclient(hunting): picked mon.noname-c con 0x7f2cf80af8e0 addr v1:172.21.15.18:6790/0 2024-03-29T11:11:19.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.474+0000 7f2d00015640 10 monclient(hunting): picked mon.noname-b con 0x7f2cf80b0860 addr v1:172.21.15.70:6789/0 2024-03-29T11:11:19.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.474+0000 7f2d00015640 10 monclient(hunting): picked mon.noname-a con 0x7f2cf813ef30 addr v1:172.21.15.18:6789/0 2024-03-29T11:11:19.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.474+0000 7f2d00015640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cf81d6d90 con 0x7f2cf813ef30 2024-03-29T11:11:19.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.474+0000 7f2d00015640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cf8136220 con 0x7f2cf80b0860 2024-03-29T11:11:19.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.476+0000 7f2d00015640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cf81363b0 con 0x7f2cf80af8e0 2024-03-29T11:11:19.455 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.476+0000 7f2d00015640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:19.455 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.476+0000 7f2cfddb1640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f2cf80af8e0 0x7f2cf81d72b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:41462/0 (socket says 172.21.15.18:41462) 2024-03-29T11:11:19.455 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.476+0000 7f2cfddb1640 1 -- 172.21.15.18:0/2755923319 learned_addr learned my addr 172.21.15.18:0/2755923319 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:11:19.457 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.476+0000 7f2d00015640 10 monclient(hunting): authenticate will time out at 905.748214s 2024-03-29T11:11:19.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.478+0000 7f2cee7fc640 10 client.?.objecter ms_handle_connect 0x7f2cf80af8e0 2024-03-29T11:11:19.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.478+0000 7f2cee7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:19.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.478+0000 7f2cee7fc640 1 -- 172.21.15.18:0/2755923319 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3205084118 0 0) 0x7f2cf81363b0 con 0x7f2cf80af8e0 2024-03-29T11:11:19.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.478+0000 7f2cee7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:11:19.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.478+0000 7f2cee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:19.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.478+0000 7f2cee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:19.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.478+0000 7f2cee7fc640 10 monclient(hunting): my global_id is 4322 2024-03-29T11:11:19.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.478+0000 7f2cee7fc640 1 -- 172.21.15.18:0/2755923319 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cdc004eb0 con 0x7f2cf80af8e0 2024-03-29T11:11:19.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.478+0000 7f2cee7fc640 10 client.?.objecter ms_handle_connect 0x7f2cf80b0860 2024-03-29T11:11:19.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.478+0000 7f2cee7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:19.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.478+0000 7f2cee7fc640 1 -- 172.21.15.18:0/2755923319 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 488410076 0 0) 0x7f2cf8136220 con 0x7f2cf80b0860 2024-03-29T11:11:19.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.479+0000 7f2cee7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:11:19.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.479+0000 7f2cee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:19.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.479+0000 7f2cee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:19.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.479+0000 7f2cee7fc640 10 monclient(hunting): my global_id is 4309 2024-03-29T11:11:19.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.479+0000 7f2cee7fc640 1 -- 172.21.15.18:0/2755923319 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cf81363b0 con 0x7f2cf80b0860 2024-03-29T11:11:19.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.479+0000 7f2cee7fc640 1 -- 172.21.15.18:0/2755923319 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1530491746 0 0) 0x7f2cf81363b0 con 0x7f2cf80b0860 2024-03-29T11:11:19.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.479+0000 7f2cee7fc640 1 -- 172.21.15.18:0/2755923319 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2cf8136220 con 0x7f2cf80b0860 2024-03-29T11:11:19.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.479+0000 7f2cee7fc640 1 -- 172.21.15.18:0/2755923319 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f2ce8001ba0 con 0x7f2cf80b0860 2024-03-29T11:11:19.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.480+0000 7f2cee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:19.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.480+0000 7f2cee7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:11:19.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.480+0000 7f2cee7fc640 10 monclient(hunting): dump: 2024-03-29T11:11:19.463 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:11:19.463 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:19.463 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:19.463 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:19.463 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:19.463 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:19.463 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:11:19.463 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:11:19.463 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:11:19.463 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:19.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.480+0000 7f2cee7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:11:19.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.480+0000 7f2cee7fc640 10 client.?.objecter ms_handle_connect 0x7f2cf813ef30 2024-03-29T11:11:19.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.480+0000 7f2cee7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:19.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.480+0000 7f2cee7fc640 1 -- 172.21.15.18:0/2755923319 <== mon.1 v1:172.21.15.70:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2854871800 0 0) 0x7f2cf8136220 con 0x7f2cf80b0860 2024-03-29T11:11:19.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.480+0000 7f2cee7fc640 1 -- 172.21.15.18:0/2755923319 >> v1:172.21.15.18:6790/0 conn(0x7f2cf80af8e0 legacy=0x7f2cf81d72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:19.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.480+0000 7f2cee7fc640 1 -- 172.21.15.18:0/2755923319 >> v1:172.21.15.18:6789/0 conn(0x7f2cf813ef30 legacy=0x7f2cf81ddce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:19.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.480+0000 7f2cee7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:11:19.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.480+0000 7f2cee7fc640 1 monclient: found mon.b 2024-03-29T11:11:19.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.480+0000 7f2cee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:19.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.480+0000 7f2cee7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:11:19.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.480+0000 7f2cee7fc640 1 -- 172.21.15.18:0/2755923319 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf81d69b0 con 0x7f2cf80b0860 2024-03-29T11:11:19.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.480+0000 7f2cee7fc640 10 monclient: _finish_auth 0 2024-03-29T11:11:19.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.480+0000 7f2cee7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:11:19.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.480+0000 7f2cee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:19.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.480+0000 7f2d00015640 5 monclient: authenticate success, global_id 4309 2024-03-29T11:11:19.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.480+0000 7f2d00015640 10 monclient: _renew_subs 2024-03-29T11:11:19.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.480+0000 7f2d00015640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:11:19.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.480+0000 7f2d00015640 1 -- 172.21.15.18:0/2755923319 --> v1:172.21.15.70:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cf8135c90 con 0x7f2cf80b0860 2024-03-29T11:11:19.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.480+0000 7f2cee7fc640 1 -- 172.21.15.18:0/2755923319 <== mon.1 v1:172.21.15.70:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ce80038a0 con 0x7f2cf80b0860 2024-03-29T11:11:19.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.480+0000 7f2cee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:19.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.480+0000 7f2d00015640 10 client.4309.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:11:19.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.480+0000 7f2d00015640 10 monclient: _renew_subs 2024-03-29T11:11:19.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.480+0000 7f2cee7fc640 1 -- 172.21.15.18:0/2755923319 <== mon.1 v1:172.21.15.70:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f2ce8003bb0 con 0x7f2cf80b0860 2024-03-29T11:11:19.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.480+0000 7f2d00015640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:11:19.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.480+0000 7f2d00015640 1 -- 172.21.15.18:0/2755923319 --> v1:172.21.15.70:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cf81df400 con 0x7f2cf80b0860 2024-03-29T11:11:19.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.480+0000 7f2d00015640 1 librados: init done 2024-03-29T11:11:19.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.480+0000 7f2cee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:19.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.480+0000 7f2cee7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:11:19.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.480+0000 7f2cee7fc640 10 monclient: dump: 2024-03-29T11:11:19.465 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:11:19.465 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:19.465 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:19.466 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:19.466 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:19.466 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:19.466 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:11:19.466 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:11:19.466 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:11:19.466 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:19.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.481+0000 7f2ceffff640 4 set_mon_vals no callback set 2024-03-29T11:11:19.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.483+0000 7f2cee7fc640 1 -- 172.21.15.18:0/2755923319 <== mon.1 v1:172.21.15.70:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f2ce8019b80 con 0x7f2cf80b0860 2024-03-29T11:11:19.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.483+0000 7f2cfddb1640 10 monclient: get_auth_request con 0x7f2cdc06db60 auth_method 0 2024-03-29T11:11:19.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.483+0000 7f2cee7fc640 1 -- 172.21.15.18:0/2755923319 <== mon.1 v1:172.21.15.70:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f2ce8083880 con 0x7f2cf80b0860 2024-03-29T11:11:19.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.483+0000 7f2cee7fc640 10 client.4309.objecter ms_dispatch 0x7f2cf8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:11:19.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.483+0000 7f2cee7fc640 3 client.4309.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:11:19.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.483+0000 7f2cee7fc640 3 client.4309.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:11:19.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.483+0000 7f2cee7fc640 20 client.4309.objecter dump_active .. 0 homeless 2024-03-29T11:11:19.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.484+0000 7f2cee7fc640 10 client.4309.objecter ms_handle_connect 0x7f2cdc06db60 2024-03-29T11:11:19.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.484+0000 7f2d00015640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:19.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.484+0000 7f2d00015640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:19.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.484+0000 7f2d00015640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:11:19.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.484+0000 7f2d00015640 1 -- 172.21.15.18:0/2755923319 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ce0005080 con 0x7f2cf80b0860 2024-03-29T11:11:19.469 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.491+0000 7f2cee7fc640 1 -- 172.21.15.18:0/2755923319 <== mon.1 v1:172.21.15.70:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2ce804f160 con 0x7f2cf80b0860 2024-03-29T11:11:19.469 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.491+0000 7f2cee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:19.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.491+0000 7f2cee7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:11:19.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.865+0000 7f2d00015640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:19.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.865+0000 7f2d00015640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:19.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.865+0000 7f2d00015640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:11:19.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.865+0000 7f2d00015640 1 -- 172.21.15.18:0/2755923319 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ce0005330 con 0x7f2cf80b0860 2024-03-29T11:11:19.847 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:11:19.847 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:11:19.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.866+0000 7f2cee7fc640 1 -- 172.21.15.18:0/2755923319 <== mon.1 v1:172.21.15.70:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f2ce8051090 con 0x7f2cf80b0860 2024-03-29T11:11:19.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.866+0000 7f2cee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:19.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.866+0000 7f2cee7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:11:19.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.869+0000 7f2cdbfff640 10 librados: watch_flush enter 2024-03-29T11:11:19.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.873+0000 7f2cdbfff640 10 librados: watch_flush exit 2024-03-29T11:11:19.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.873+0000 7f2cdbfff640 20 client.4309.objecter shutdown clearing up homeless session... 2024-03-29T11:11:19.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.873+0000 7f2cdbfff640 10 client.4309.objecter successfully canceled tick 2024-03-29T11:11:19.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.873+0000 7f2cdbfff640 1 -- 172.21.15.18:0/2755923319 >> v1:172.21.15.70:6816/3101871562 conn(0x7f2cdc06db60 legacy=0x7f2cdc070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:19.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.874+0000 7f2cdbfff640 10 monclient: shutdown 2024-03-29T11:11:19.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.874+0000 7f2cdbfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:19.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.874+0000 7f2cdbfff640 1 -- 172.21.15.18:0/2755923319 >> v1:172.21.15.70:6789/0 conn(0x7f2cf80b0860 legacy=0x7f2cf8135580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:19.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.874+0000 7f2cdbfff640 1 -- 172.21.15.18:0/2755923319 shutdown_connections 2024-03-29T11:11:19.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.874+0000 7f2cdbfff640 1 -- 172.21.15.18:0/2755923319 >> 172.21.15.18:0/2755923319 conn(0x7f2cf806b660 msgr2=0x7f2cf80ac870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:19.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.874+0000 7f2cdbfff640 1 -- 172.21.15.18:0/2755923319 shutdown_connections 2024-03-29T11:11:19.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.874+0000 7f2cdbfff640 1 -- 172.21.15.18:0/2755923319 wait complete. 2024-03-29T11:11:19.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:19.874+0000 7f2cdbfff640 1 librados: shutdown 2024-03-29T11:11:19.869 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:11:26.676 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:11:26.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.842+0000 7f4473193640 10 monclient: get_monmap_and_config 2024-03-29T11:11:26.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.842+0000 7f4473193640 10 monclient: build_initial_monmap 2024-03-29T11:11:26.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.842+0000 7f4473193640 10 monclient: monmap: 2024-03-29T11:11:26.821 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:11:26.821 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:26.821 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:11:26.843845+0000 2024-03-29T11:11:26.821 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:11:26.843845+0000 2024-03-29T11:11:26.821 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:26.821 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:26.821 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:11:26.821 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:11:26.821 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:11:26.821 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:26.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.843+0000 7f4473193640 1 Processor -- start 2024-03-29T11:11:26.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.843+0000 7f4473193640 1 -- start start 2024-03-29T11:11:26.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.844+0000 7f4473193640 10 monclient: init 2024-03-29T11:11:26.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.845+0000 7f4473193640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:26.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.845+0000 7f4473193640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:11:26.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.845+0000 7f4473193640 10 monclient(hunting): picked mon.noname-a con 0x7f446c1457f0 addr v1:172.21.15.18:6789/0 2024-03-29T11:11:26.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.845+0000 7f4473193640 10 monclient(hunting): picked mon.noname-b con 0x7f446c13f650 addr v1:172.21.15.18:6790/0 2024-03-29T11:11:26.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.845+0000 7f4470f2f640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f446c1457f0 0x7f446c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:46992/0 (socket says 172.21.15.18:46992) 2024-03-29T11:11:26.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.845+0000 7f4473193640 10 monclient(hunting): picked mon.noname-c con 0x7f446c0a9d70 addr v1:172.21.15.70:6789/0 2024-03-29T11:11:26.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.845+0000 7f4470f2f640 1 -- 172.21.15.18:0/109038749 learned_addr learned my addr 172.21.15.18:0/109038749 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:11:26.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.845+0000 7f4473193640 1 -- 172.21.15.18:0/109038749 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f446c0aaef0 con 0x7f446c1457f0 2024-03-29T11:11:26.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.845+0000 7f4473193640 1 -- 172.21.15.18:0/109038749 --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f446c145bd0 con 0x7f446c0a9d70 2024-03-29T11:11:26.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.845+0000 7f4473193640 1 -- 172.21.15.18:0/109038749 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f446c145eb0 con 0x7f446c13f650 2024-03-29T11:11:26.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.845+0000 7f4473193640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:26.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.845+0000 7f4473193640 10 monclient(hunting): authenticate will time out at 913.117586s 2024-03-29T11:11:26.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.845+0000 7f446b7fe640 1 -- 172.21.15.18:0/109038749 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2560690104 0 0) 0x7f446c0aaef0 con 0x7f446c1457f0 2024-03-29T11:11:26.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.846+0000 7f446b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:11:26.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.846+0000 7f446b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:26.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.846+0000 7f446b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:26.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.846+0000 7f446b7fe640 10 monclient(hunting): my global_id is 4308 2024-03-29T11:11:26.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.846+0000 7f446b7fe640 1 -- 172.21.15.18:0/109038749 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f445c003520 con 0x7f446c1457f0 2024-03-29T11:11:26.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.846+0000 7f446b7fe640 1 -- 172.21.15.18:0/109038749 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1131024184 0 0) 0x7f446c145bd0 con 0x7f446c0a9d70 2024-03-29T11:11:26.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.846+0000 7f446b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:11:26.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.846+0000 7f446b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:26.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.846+0000 7f446b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:26.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.846+0000 7f446b7fe640 10 monclient(hunting): my global_id is 4312 2024-03-29T11:11:26.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.846+0000 7f446b7fe640 1 -- 172.21.15.18:0/109038749 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f446c0aaef0 con 0x7f446c0a9d70 2024-03-29T11:11:26.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.846+0000 7f446b7fe640 1 -- 172.21.15.18:0/109038749 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3781498737 0 0) 0x7f445c003520 con 0x7f446c1457f0 2024-03-29T11:11:26.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.846+0000 7f446b7fe640 1 -- 172.21.15.18:0/109038749 >> v1:172.21.15.18:6790/0 conn(0x7f446c13f650 legacy=0x7f446c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:26.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.846+0000 7f446b7fe640 1 -- 172.21.15.18:0/109038749 >> v1:172.21.15.70:6789/0 conn(0x7f446c0a9d70 legacy=0x7f446c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:26.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.846+0000 7f446b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:11:26.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.846+0000 7f446b7fe640 1 monclient: found mon.noname-a 2024-03-29T11:11:26.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.846+0000 7f446b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:26.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.846+0000 7f446b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-29T11:11:26.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.846+0000 7f446b7fe640 1 -- 172.21.15.18:0/109038749 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f446c146340 con 0x7f446c1457f0 2024-03-29T11:11:26.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.846+0000 7f446b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:11:26.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.846+0000 7f446b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:11:26.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.846+0000 7f446b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:26.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.846+0000 7f446b7fe640 1 -- 172.21.15.18:0/109038749 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f4460001690 con 0x7f446c1457f0 2024-03-29T11:11:26.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.846+0000 7f446b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:26.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.846+0000 7f446b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:11:26.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.846+0000 7f446b7fe640 10 monclient: dump: 2024-03-29T11:11:26.827 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:11:26.827 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:26.827 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:26.827 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:26.827 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:26.827 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:26.827 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:11:26.827 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:11:26.827 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:11:26.827 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:26.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.846+0000 7f4473193640 5 monclient: authenticate success, global_id 4308 2024-03-29T11:11:26.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.846+0000 7f4473193640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:26.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.846+0000 7f446b7fe640 1 -- 172.21.15.18:0/109038749 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44600017f0 con 0x7f446c1457f0 2024-03-29T11:11:26.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.846+0000 7f446b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:26.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.846+0000 7f446b7fe640 1 -- 172.21.15.18:0/109038749 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f4460001c00 con 0x7f446c1457f0 2024-03-29T11:11:26.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.846+0000 7f446b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:26.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.846+0000 7f446b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:11:26.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.846+0000 7f446b7fe640 10 monclient: dump: 2024-03-29T11:11:26.828 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:11:26.828 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:26.828 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:26.828 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:26.828 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:26.829 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:26.829 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:11:26.829 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:11:26.829 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:11:26.829 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:26.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.847+0000 7f4473193640 10 monclient: get_monmap_and_config success 2024-03-29T11:11:26.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.847+0000 7f4473193640 4 set_mon_vals no callback set 2024-03-29T11:11:26.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.847+0000 7f4473193640 10 monclient: shutdown 2024-03-29T11:11:26.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.847+0000 7f4473193640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:26.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.847+0000 7f4473193640 1 -- 172.21.15.18:0/109038749 >> v1:172.21.15.18:6789/0 conn(0x7f446c1457f0 legacy=0x7f446c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:26.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.848+0000 7f4473193640 1 -- 172.21.15.18:0/109038749 shutdown_connections 2024-03-29T11:11:26.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.848+0000 7f4473193640 1 -- 172.21.15.18:0/109038749 >> 172.21.15.18:0/109038749 conn(0x7f446c066a00 msgr2=0x7f446c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:26.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.848+0000 7f4473193640 1 -- 172.21.15.18:0/109038749 shutdown_connections 2024-03-29T11:11:26.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.848+0000 7f4473193640 1 -- 172.21.15.18:0/109038749 wait complete. 2024-03-29T11:11:26.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.849+0000 7f4473193640 10 monclient: build_initial_monmap 2024-03-29T11:11:26.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.849+0000 7f4473193640 10 monclient: monmap: 2024-03-29T11:11:26.840 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:11:26.840 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:11:26.840 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:11:26.840 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:11:26.840 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:26.840 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:26.840 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:11:26.840 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:11:26.840 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:11:26.840 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:26.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.850+0000 7f4473193640 1 librados: starting msgr at 2024-03-29T11:11:26.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.850+0000 7f4473193640 1 librados: starting objecter 2024-03-29T11:11:26.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.850+0000 7f4473193640 1 Processor -- start 2024-03-29T11:11:26.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.850+0000 7f4473193640 1 -- start start 2024-03-29T11:11:26.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.850+0000 7f4473193640 1 librados: setting wanted keys 2024-03-29T11:11:26.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.850+0000 7f4473193640 1 librados: calling monclient init 2024-03-29T11:11:26.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.850+0000 7f4473193640 10 monclient: init 2024-03-29T11:11:26.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.851+0000 7f4473193640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:26.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.851+0000 7f4473193640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:11:26.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.851+0000 7f4473193640 10 monclient(hunting): picked mon.noname-c con 0x7f446c0a9d70 addr v1:172.21.15.18:6790/0 2024-03-29T11:11:26.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.851+0000 7f4473193640 10 monclient(hunting): picked mon.noname-b con 0x7f446c13f650 addr v1:172.21.15.70:6789/0 2024-03-29T11:11:26.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.851+0000 7f4473193640 10 monclient(hunting): picked mon.noname-a con 0x7f446c1457f0 addr v1:172.21.15.18:6789/0 2024-03-29T11:11:26.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.851+0000 7f4473193640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f446c1dc1c0 con 0x7f446c1457f0 2024-03-29T11:11:26.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.851+0000 7f4473193640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f446c1dcd60 con 0x7f446c13f650 2024-03-29T11:11:26.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.851+0000 7f4470f2f640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f446c0a9d70 0x7f446c06e100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:37016/0 (socket says 172.21.15.18:37016) 2024-03-29T11:11:26.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.851+0000 7f4470f2f640 1 -- 172.21.15.18:0/1055303213 learned_addr learned my addr 172.21.15.18:0/1055303213 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:11:26.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.851+0000 7f44697fa640 10 client.?.objecter ms_handle_connect 0x7f446c0a9d70 2024-03-29T11:11:26.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.851+0000 7f44697fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:26.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.851+0000 7f4473193640 1 -- 172.21.15.18:0/1055303213 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f446c06fd10 con 0x7f446c0a9d70 2024-03-29T11:11:26.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.851+0000 7f4473193640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:26.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.851+0000 7f4473193640 10 monclient(hunting): authenticate will time out at 913.123654s 2024-03-29T11:11:26.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.852+0000 7f44697fa640 10 client.?.objecter ms_handle_connect 0x7f446c13f650 2024-03-29T11:11:26.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.852+0000 7f44697fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:26.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.852+0000 7f44697fa640 1 -- 172.21.15.18:0/1055303213 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3055736932 0 0) 0x7f446c1dcd60 con 0x7f446c13f650 2024-03-29T11:11:26.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.852+0000 7f44697fa640 10 monclient(hunting): _negotiate 2024-03-29T11:11:26.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.852+0000 7f44697fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:26.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.852+0000 7f44697fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:26.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.852+0000 7f44697fa640 10 monclient(hunting): my global_id is 4315 2024-03-29T11:11:26.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.852+0000 7f44697fa640 1 -- 172.21.15.18:0/1055303213 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4450004eb0 con 0x7f446c13f650 2024-03-29T11:11:26.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.852+0000 7f44697fa640 1 -- 172.21.15.18:0/1055303213 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1553231798 0 0) 0x7f446c06fd10 con 0x7f446c0a9d70 2024-03-29T11:11:26.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.852+0000 7f44697fa640 10 monclient(hunting): _negotiate 2024-03-29T11:11:26.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.852+0000 7f44697fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:26.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.852+0000 7f44697fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:26.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.852+0000 7f44697fa640 10 monclient(hunting): my global_id is 4328 2024-03-29T11:11:26.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.852+0000 7f44697fa640 1 -- 172.21.15.18:0/1055303213 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f446c1dcd60 con 0x7f446c0a9d70 2024-03-29T11:11:26.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.852+0000 7f44697fa640 10 client.?.objecter ms_handle_connect 0x7f446c1457f0 2024-03-29T11:11:26.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.852+0000 7f44697fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:26.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.852+0000 7f44697fa640 1 -- 172.21.15.18:0/1055303213 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1462120014 0 0) 0x7f4450004eb0 con 0x7f446c13f650 2024-03-29T11:11:26.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.852+0000 7f44697fa640 1 -- 172.21.15.18:0/1055303213 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f446c06fd10 con 0x7f446c13f650 2024-03-29T11:11:26.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.852+0000 7f44697fa640 1 -- 172.21.15.18:0/1055303213 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f44580018c0 con 0x7f446c13f650 2024-03-29T11:11:26.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.852+0000 7f44697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:26.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.852+0000 7f44697fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:11:26.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.852+0000 7f44697fa640 10 monclient(hunting): dump: 2024-03-29T11:11:26.843 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:11:26.843 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:26.843 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:26.844 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:26.844 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:26.844 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:26.844 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:11:26.844 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:11:26.844 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:11:26.844 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:26.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.852+0000 7f44697fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:11:26.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.852+0000 7f44697fa640 1 -- 172.21.15.18:0/1055303213 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1904506314 0 0) 0x7f446c1dc1c0 con 0x7f446c1457f0 2024-03-29T11:11:26.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.852+0000 7f44697fa640 10 monclient(hunting): _negotiate 2024-03-29T11:11:26.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.852+0000 7f44697fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:26.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.852+0000 7f44697fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:26.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.852+0000 7f44697fa640 10 monclient(hunting): my global_id is 4311 2024-03-29T11:11:26.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.852+0000 7f44697fa640 1 -- 172.21.15.18:0/1055303213 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4450004eb0 con 0x7f446c1457f0 2024-03-29T11:11:26.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.853+0000 7f44697fa640 1 -- 172.21.15.18:0/1055303213 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 136692131 0 0) 0x7f446c1dcd60 con 0x7f446c0a9d70 2024-03-29T11:11:26.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.853+0000 7f44697fa640 1 -- 172.21.15.18:0/1055303213 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f446c1dc1c0 con 0x7f446c0a9d70 2024-03-29T11:11:26.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.853+0000 7f44697fa640 1 -- 172.21.15.18:0/1055303213 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f4460001c00 con 0x7f446c0a9d70 2024-03-29T11:11:26.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.853+0000 7f44697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:26.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.853+0000 7f44697fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:26.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.853+0000 7f44697fa640 10 monclient(hunting): dump: 2024-03-29T11:11:26.845 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:11:26.845 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:26.845 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:26.845 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:26.845 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:26.845 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:26.845 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:11:26.845 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:11:26.845 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:11:26.846 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:26.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.853+0000 7f44697fa640 1 -- 172.21.15.18:0/1055303213 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1488878692 0 0) 0x7f446c1dc1c0 con 0x7f446c0a9d70 2024-03-29T11:11:26.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.853+0000 7f44697fa640 1 -- 172.21.15.18:0/1055303213 >> v1:172.21.15.70:6789/0 conn(0x7f446c13f650 legacy=0x7f446c1dc650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:26.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.853+0000 7f44697fa640 1 -- 172.21.15.18:0/1055303213 >> v1:172.21.15.18:6789/0 conn(0x7f446c1457f0 legacy=0x7f446c137780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:26.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.853+0000 7f44697fa640 10 monclient: _finish_hunting 0 2024-03-29T11:11:26.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.853+0000 7f44697fa640 1 monclient: found mon.c 2024-03-29T11:11:26.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.853+0000 7f44697fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:26.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.853+0000 7f44697fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:11:26.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.853+0000 7f44697fa640 1 -- 172.21.15.18:0/1055303213 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f446c1dbe50 con 0x7f446c0a9d70 2024-03-29T11:11:26.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.853+0000 7f44697fa640 10 monclient: _finish_auth 0 2024-03-29T11:11:26.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.853+0000 7f44697fa640 10 monclient: _check_auth_tickets 2024-03-29T11:11:26.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.853+0000 7f44697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:26.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.853+0000 7f4473193640 5 monclient: authenticate success, global_id 4328 2024-03-29T11:11:26.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.853+0000 7f4473193640 10 monclient: _renew_subs 2024-03-29T11:11:26.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.853+0000 7f4473193640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:11:26.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.853+0000 7f4473193640 1 -- 172.21.15.18:0/1055303213 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f446c06e930 con 0x7f446c0a9d70 2024-03-29T11:11:26.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.853+0000 7f4473193640 10 client.4328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:11:26.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.853+0000 7f4473193640 10 monclient: _renew_subs 2024-03-29T11:11:26.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.853+0000 7f4473193640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:11:26.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.853+0000 7f4473193640 1 -- 172.21.15.18:0/1055303213 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f446c138070 con 0x7f446c0a9d70 2024-03-29T11:11:26.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.853+0000 7f4473193640 1 librados: init done 2024-03-29T11:11:26.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.855+0000 7f4473193640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:26.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.855+0000 7f4473193640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:26.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.855+0000 7f4473193640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:11:26.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.855+0000 7f4473193640 1 -- 172.21.15.18:0/1055303213 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f446c06fb50 con 0x7f446c0a9d70 2024-03-29T11:11:26.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.855+0000 7f44697fa640 1 -- 172.21.15.18:0/1055303213 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4460004a90 con 0x7f446c0a9d70 2024-03-29T11:11:26.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.855+0000 7f44697fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:26.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.855+0000 7f44697fa640 1 -- 172.21.15.18:0/1055303213 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f4460004da0 con 0x7f446c0a9d70 2024-03-29T11:11:26.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.855+0000 7f44697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:26.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.855+0000 7f446affd640 4 set_mon_vals no callback set 2024-03-29T11:11:26.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.855+0000 7f44697fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:26.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.855+0000 7f44697fa640 10 monclient: dump: 2024-03-29T11:11:26.848 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:11:26.848 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:26.848 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:26.848 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:26.848 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:26.848 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:26.848 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:11:26.848 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:11:26.848 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:11:26.848 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:26.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.856+0000 7f44697fa640 1 -- 172.21.15.18:0/1055303213 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f446001ad70 con 0x7f446c0a9d70 2024-03-29T11:11:26.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.857+0000 7f44697fa640 1 -- 172.21.15.18:0/1055303213 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f4460084ae0 con 0x7f446c0a9d70 2024-03-29T11:11:26.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.857+0000 7f44697fa640 10 client.4328.objecter ms_dispatch 0x7f446c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:11:26.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.857+0000 7f44697fa640 3 client.4328.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:11:26.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.857+0000 7f44697fa640 3 client.4328.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:11:26.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.857+0000 7f446bfff640 10 monclient: get_auth_request con 0x7f445006df00 auth_method 0 2024-03-29T11:11:26.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.857+0000 7f44697fa640 20 client.4328.objecter dump_active .. 0 homeless 2024-03-29T11:11:26.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.857+0000 7f44697fa640 10 client.4328.objecter ms_handle_connect 0x7f445006df00 2024-03-29T11:11:26.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.863+0000 7f44697fa640 1 -- 172.21.15.18:0/1055303213 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4460050420 con 0x7f446c0a9d70 2024-03-29T11:11:26.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.863+0000 7f44697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:26.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:26.863+0000 7f44697fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:11:27.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:27.247+0000 7f4473193640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:27.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:27.247+0000 7f4473193640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:27.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:27.247+0000 7f4473193640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:11:27.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:27.247+0000 7f4473193640 1 -- 172.21.15.18:0/1055303213 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f446c075f10 con 0x7f446c0a9d70 2024-03-29T11:11:27.227 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:27.249+0000 7f44697fa640 1 -- 172.21.15.18:0/1055303213 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f4460084190 con 0x7f446c0a9d70 2024-03-29T11:11:27.227 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:27.249+0000 7f44697fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:27.227 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:27.249+0000 7f44697fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:11:27.228 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:11:27.228 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:11:27.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:27.252+0000 7f4473193640 10 librados: watch_flush enter 2024-03-29T11:11:27.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:27.253+0000 7f4473193640 10 librados: watch_flush exit 2024-03-29T11:11:27.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:27.253+0000 7f4473193640 20 client.4328.objecter shutdown clearing up homeless session... 2024-03-29T11:11:27.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:27.253+0000 7f4473193640 10 client.4328.objecter successfully canceled tick 2024-03-29T11:11:27.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:27.253+0000 7f4473193640 1 -- 172.21.15.18:0/1055303213 >> v1:172.21.15.70:6816/3101871562 conn(0x7f445006df00 legacy=0x7f44500703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:27.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:27.253+0000 7f4473193640 10 monclient: shutdown 2024-03-29T11:11:27.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:27.253+0000 7f4473193640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:27.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:27.253+0000 7f4473193640 1 -- 172.21.15.18:0/1055303213 >> v1:172.21.15.18:6790/0 conn(0x7f446c0a9d70 legacy=0x7f446c06e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:27.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:27.253+0000 7f4473193640 1 -- 172.21.15.18:0/1055303213 shutdown_connections 2024-03-29T11:11:27.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:27.253+0000 7f4473193640 1 -- 172.21.15.18:0/1055303213 >> 172.21.15.18:0/1055303213 conn(0x7f446c066a00 msgr2=0x7f446c13dcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:27.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:27.253+0000 7f4473193640 1 -- 172.21.15.18:0/1055303213 shutdown_connections 2024-03-29T11:11:27.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:27.253+0000 7f4473193640 1 -- 172.21.15.18:0/1055303213 wait complete. 2024-03-29T11:11:27.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:27.253+0000 7f4473193640 1 librados: shutdown 2024-03-29T11:11:27.245 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:11:34.053 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:11:34.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.211+0000 7f1c954e6640 10 monclient: get_monmap_and_config 2024-03-29T11:11:34.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.211+0000 7f1c954e6640 10 monclient: build_initial_monmap 2024-03-29T11:11:34.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.211+0000 7f1c954e6640 10 monclient: monmap: 2024-03-29T11:11:34.190 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:11:34.190 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:34.190 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:11:34.212029+0000 2024-03-29T11:11:34.190 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:11:34.212029+0000 2024-03-29T11:11:34.190 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:34.190 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:34.190 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:11:34.190 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:11:34.191 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:11:34.191 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:34.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.212+0000 7f1c954e6640 1 Processor -- start 2024-03-29T11:11:34.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.212+0000 7f1c954e6640 1 -- start start 2024-03-29T11:11:34.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.212+0000 7f1c954e6640 10 monclient: init 2024-03-29T11:11:34.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.212+0000 7f1c954e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:34.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.212+0000 7f1c954e6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:11:34.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.212+0000 7f1c954e6640 10 monclient(hunting): picked mon.noname-c con 0x7f1c900a9ac0 addr v1:172.21.15.70:6789/0 2024-03-29T11:11:34.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.212+0000 7f1c954e6640 10 monclient(hunting): picked mon.noname-b con 0x7f1c9013ee90 addr v1:172.21.15.18:6790/0 2024-03-29T11:11:34.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.212+0000 7f1c954e6640 10 monclient(hunting): picked mon.noname-a con 0x7f1c900af900 addr v1:172.21.15.18:6789/0 2024-03-29T11:11:34.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.212+0000 7f1c954e6640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c900afd10 con 0x7f1c900af900 2024-03-29T11:11:34.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.212+0000 7f1c954e6640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c900af6f0 con 0x7f1c900a9ac0 2024-03-29T11:11:34.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.212+0000 7f1c954e6640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c900aac80 con 0x7f1c9013ee90 2024-03-29T11:11:34.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.212+0000 7f1c954e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:34.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.212+0000 7f1c954e6640 10 monclient(hunting): authenticate will time out at 920.484864s 2024-03-29T11:11:34.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.213+0000 7f1c8e7fc640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f1c9013ee90 0x7f1c9013b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:48430/0 (socket says 172.21.15.18:48430) 2024-03-29T11:11:34.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.213+0000 7f1c8e7fc640 1 -- 172.21.15.18:0/3721016581 learned_addr learned my addr 172.21.15.18:0/3721016581 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:11:34.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.213+0000 7f1c8dffb640 1 -- 172.21.15.18:0/3721016581 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3845995075 0 0) 0x7f1c900aac80 con 0x7f1c9013ee90 2024-03-29T11:11:34.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.213+0000 7f1c8dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:11:34.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.213+0000 7f1c8dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:34.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.213+0000 7f1c8dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:34.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.213+0000 7f1c8dffb640 10 monclient(hunting): my global_id is 4331 2024-03-29T11:11:34.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.213+0000 7f1c8dffb640 1 -- 172.21.15.18:0/3721016581 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c78003520 con 0x7f1c9013ee90 2024-03-29T11:11:34.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.213+0000 7f1c8dffb640 1 -- 172.21.15.18:0/3721016581 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 659559019 0 0) 0x7f1c78003520 con 0x7f1c9013ee90 2024-03-29T11:11:34.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.213+0000 7f1c8dffb640 1 -- 172.21.15.18:0/3721016581 >> v1:172.21.15.70:6789/0 conn(0x7f1c900a9ac0 legacy=0x7f1c900adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:34.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.213+0000 7f1c8dffb640 1 -- 172.21.15.18:0/3721016581 >> v1:172.21.15.18:6789/0 conn(0x7f1c900af900 legacy=0x7f1c90149b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:11:34.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.214+0000 7f1c8dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:11:34.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.214+0000 7f1c8dffb640 1 monclient: found mon.noname-b 2024-03-29T11:11:34.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.214+0000 7f1c8dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:34.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.214+0000 7f1c8dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:11:34.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.214+0000 7f1c8dffb640 1 -- 172.21.15.18:0/3721016581 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c900aaf60 con 0x7f1c9013ee90 2024-03-29T11:11:34.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.214+0000 7f1c8dffb640 10 monclient: _finish_auth 0 2024-03-29T11:11:34.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.214+0000 7f1c8dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:11:34.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.214+0000 7f1c8dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:34.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.214+0000 7f1c8dffb640 1 -- 172.21.15.18:0/3721016581 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f1c84001690 con 0x7f1c9013ee90 2024-03-29T11:11:34.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.214+0000 7f1c8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:34.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.215+0000 7f1c8dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:11:34.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.215+0000 7f1c8dffb640 10 monclient: dump: 2024-03-29T11:11:34.201 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:11:34.201 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:34.201 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:34.201 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:34.201 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:34.201 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:34.202 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:11:34.202 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:11:34.202 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:11:34.202 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:34.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.215+0000 7f1c954e6640 5 monclient: authenticate success, global_id 4331 2024-03-29T11:11:34.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.215+0000 7f1c954e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:34.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.215+0000 7f1c8dffb640 1 -- 172.21.15.18:0/3721016581 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c84001860 con 0x7f1c9013ee90 2024-03-29T11:11:34.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.215+0000 7f1c8dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:34.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.215+0000 7f1c8dffb640 1 -- 172.21.15.18:0/3721016581 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f1c84001c70 con 0x7f1c9013ee90 2024-03-29T11:11:34.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.215+0000 7f1c8dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:34.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.215+0000 7f1c8dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:34.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.215+0000 7f1c8dffb640 10 monclient: dump: 2024-03-29T11:11:34.202 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:11:34.202 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:34.203 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:34.203 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:34.203 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:34.203 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:34.203 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:11:34.203 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:11:34.203 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:11:34.203 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:34.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.215+0000 7f1c954e6640 10 monclient: get_monmap_and_config success 2024-03-29T11:11:34.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.215+0000 7f1c954e6640 4 set_mon_vals no callback set 2024-03-29T11:11:34.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.216+0000 7f1c954e6640 10 monclient: shutdown 2024-03-29T11:11:34.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.216+0000 7f1c954e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:34.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.216+0000 7f1c954e6640 1 -- 172.21.15.18:0/3721016581 >> v1:172.21.15.18:6790/0 conn(0x7f1c9013ee90 legacy=0x7f1c9013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:34.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.216+0000 7f1c954e6640 1 -- 172.21.15.18:0/3721016581 shutdown_connections 2024-03-29T11:11:34.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.216+0000 7f1c954e6640 1 -- 172.21.15.18:0/3721016581 >> 172.21.15.18:0/3721016581 conn(0x7f1c900b35e0 msgr2=0x7f1c900b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:34.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.217+0000 7f1c954e6640 1 -- 172.21.15.18:0/3721016581 shutdown_connections 2024-03-29T11:11:34.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.217+0000 7f1c954e6640 1 -- 172.21.15.18:0/3721016581 wait complete. 2024-03-29T11:11:34.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.218+0000 7f1c954e6640 10 monclient: build_initial_monmap 2024-03-29T11:11:34.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.218+0000 7f1c954e6640 10 monclient: monmap: 2024-03-29T11:11:34.204 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:11:34.204 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:11:34.204 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:11:34.204 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:11:34.204 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:34.204 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:34.204 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:11:34.204 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:11:34.204 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:11:34.204 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:34.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.218+0000 7f1c954e6640 1 librados: starting msgr at 2024-03-29T11:11:34.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.218+0000 7f1c954e6640 1 librados: starting objecter 2024-03-29T11:11:34.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.219+0000 7f1c954e6640 1 Processor -- start 2024-03-29T11:11:34.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.219+0000 7f1c954e6640 1 -- start start 2024-03-29T11:11:34.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.219+0000 7f1c954e6640 1 librados: setting wanted keys 2024-03-29T11:11:34.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.219+0000 7f1c954e6640 1 librados: calling monclient init 2024-03-29T11:11:34.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.219+0000 7f1c954e6640 10 monclient: init 2024-03-29T11:11:34.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.220+0000 7f1c954e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:34.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.220+0000 7f1c954e6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:11:34.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.220+0000 7f1c954e6640 10 monclient(hunting): picked mon.noname-b con 0x7f1c900a9ac0 addr v1:172.21.15.70:6789/0 2024-03-29T11:11:34.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.220+0000 7f1c954e6640 10 monclient(hunting): picked mon.noname-c con 0x7f1c900af900 addr v1:172.21.15.18:6790/0 2024-03-29T11:11:34.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.221+0000 7f1c954e6640 10 monclient(hunting): picked mon.noname-a con 0x7f1c9013ee90 addr v1:172.21.15.18:6789/0 2024-03-29T11:11:34.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.221+0000 7f1c954e6640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c90140d20 con 0x7f1c9013ee90 2024-03-29T11:11:34.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.221+0000 7f1c954e6640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c90141c60 con 0x7f1c900a9ac0 2024-03-29T11:11:34.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.221+0000 7f1c954e6640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c901420d0 con 0x7f1c900af900 2024-03-29T11:11:34.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.221+0000 7f1c954e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:34.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.221+0000 7f1c8e7fc640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f1c900af900 0x7f1c901db910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:48646/0 (socket says 172.21.15.18:48646) 2024-03-29T11:11:34.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.221+0000 7f1c954e6640 10 monclient(hunting): authenticate will time out at 920.493349s 2024-03-29T11:11:34.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.221+0000 7f1c8e7fc640 1 -- 172.21.15.18:0/671625493 learned_addr learned my addr 172.21.15.18:0/671625493 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:11:34.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.221+0000 7f1c777fe640 10 client.?.objecter ms_handle_connect 0x7f1c900af900 2024-03-29T11:11:34.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.221+0000 7f1c777fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:34.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.221+0000 7f1c777fe640 1 -- 172.21.15.18:0/671625493 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4088232625 0 0) 0x7f1c901420d0 con 0x7f1c900af900 2024-03-29T11:11:34.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.221+0000 7f1c777fe640 10 monclient(hunting): _negotiate 2024-03-29T11:11:34.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.221+0000 7f1c777fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:34.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.221+0000 7f1c777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:34.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.221+0000 7f1c777fe640 10 monclient(hunting): my global_id is 4334 2024-03-29T11:11:34.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.221+0000 7f1c777fe640 1 -- 172.21.15.18:0/671625493 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c7c004eb0 con 0x7f1c900af900 2024-03-29T11:11:34.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.222+0000 7f1c777fe640 1 -- 172.21.15.18:0/671625493 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3392794626 0 0) 0x7f1c7c004eb0 con 0x7f1c900af900 2024-03-29T11:11:34.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.222+0000 7f1c777fe640 1 -- 172.21.15.18:0/671625493 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c901420d0 con 0x7f1c900af900 2024-03-29T11:11:34.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.222+0000 7f1c777fe640 1 -- 172.21.15.18:0/671625493 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f1c84001c30 con 0x7f1c900af900 2024-03-29T11:11:34.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.222+0000 7f1c777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:34.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.222+0000 7f1c777fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:11:34.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.222+0000 7f1c777fe640 10 monclient(hunting): dump: 2024-03-29T11:11:34.209 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:11:34.209 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:34.209 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:34.209 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:34.209 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:34.209 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:34.209 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:11:34.210 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:11:34.210 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:11:34.210 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:34.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.222+0000 7f1c777fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:11:34.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.223+0000 7f1c777fe640 10 client.?.objecter ms_handle_connect 0x7f1c9013ee90 2024-03-29T11:11:34.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.223+0000 7f1c777fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:34.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.223+0000 7f1c777fe640 1 -- 172.21.15.18:0/671625493 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2427135867 0 0) 0x7f1c90140d20 con 0x7f1c9013ee90 2024-03-29T11:11:34.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.223+0000 7f1c777fe640 10 monclient(hunting): _negotiate 2024-03-29T11:11:34.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.223+0000 7f1c777fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:34.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.223+0000 7f1c777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:34.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.223+0000 7f1c777fe640 10 monclient(hunting): my global_id is 4314 2024-03-29T11:11:34.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.223+0000 7f1c777fe640 1 -- 172.21.15.18:0/671625493 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c7c004eb0 con 0x7f1c9013ee90 2024-03-29T11:11:34.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.223+0000 7f1c777fe640 1 -- 172.21.15.18:0/671625493 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2921549023 0 0) 0x7f1c901420d0 con 0x7f1c900af900 2024-03-29T11:11:34.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.223+0000 7f1c777fe640 1 -- 172.21.15.18:0/671625493 >> v1:172.21.15.70:6789/0 conn(0x7f1c900a9ac0 legacy=0x7f1c9012f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:34.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.223+0000 7f1c777fe640 1 -- 172.21.15.18:0/671625493 >> v1:172.21.15.18:6789/0 conn(0x7f1c9013ee90 legacy=0x7f1c901dc020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:34.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.223+0000 7f1c777fe640 10 monclient: _finish_hunting 0 2024-03-29T11:11:34.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.223+0000 7f1c777fe640 1 monclient: found mon.c 2024-03-29T11:11:34.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.223+0000 7f1c777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:34.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.223+0000 7f1c777fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:11:34.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.223+0000 7f1c777fe640 1 -- 172.21.15.18:0/671625493 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c90142260 con 0x7f1c900af900 2024-03-29T11:11:34.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.223+0000 7f1c777fe640 10 monclient: _finish_auth 0 2024-03-29T11:11:34.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.223+0000 7f1c777fe640 10 monclient: _check_auth_tickets 2024-03-29T11:11:34.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.223+0000 7f1c777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:34.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.223+0000 7f1c777fe640 10 client.?.objecter ms_handle_connect 0x7f1c900a9ac0 2024-03-29T11:11:34.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.223+0000 7f1c777fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:34.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.223+0000 7f1c954e6640 5 monclient: authenticate success, global_id 4334 2024-03-29T11:11:34.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.223+0000 7f1c954e6640 10 monclient: _renew_subs 2024-03-29T11:11:34.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.223+0000 7f1c954e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:11:34.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.223+0000 7f1c954e6640 1 -- 172.21.15.18:0/671625493 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c9012f810 con 0x7f1c900af900 2024-03-29T11:11:34.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.224+0000 7f1c777fe640 1 -- 172.21.15.18:0/671625493 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c84004a90 con 0x7f1c900af900 2024-03-29T11:11:34.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.224+0000 7f1c777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:34.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.224+0000 7f1c777fe640 1 -- 172.21.15.18:0/671625493 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f1c84004ee0 con 0x7f1c900af900 2024-03-29T11:11:34.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.224+0000 7f1c777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:34.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.224+0000 7f1c777fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:34.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.224+0000 7f1c777fe640 10 monclient: dump: 2024-03-29T11:11:34.212 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:11:34.212 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:34.212 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:34.212 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:34.212 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:34.212 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:34.213 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:11:34.213 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:11:34.213 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:11:34.213 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:34.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.224+0000 7f1c954e6640 10 client.4334.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:11:34.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.224+0000 7f1c954e6640 10 monclient: _renew_subs 2024-03-29T11:11:34.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.224+0000 7f1c954e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:11:34.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.224+0000 7f1c954e6640 1 -- 172.21.15.18:0/671625493 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c90141a10 con 0x7f1c900af900 2024-03-29T11:11:34.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.224+0000 7f1c954e6640 1 librados: init done 2024-03-29T11:11:34.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.224+0000 7f1c8cff9640 4 set_mon_vals no callback set 2024-03-29T11:11:34.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.225+0000 7f1c954e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:34.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.225+0000 7f1c954e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:34.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.225+0000 7f1c954e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:11:34.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.225+0000 7f1c954e6640 1 -- 172.21.15.18:0/671625493 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c90075f10 con 0x7f1c900af900 2024-03-29T11:11:34.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.226+0000 7f1c777fe640 1 -- 172.21.15.18:0/671625493 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f1c840050c0 con 0x7f1c900af900 2024-03-29T11:11:34.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.226+0000 7f1c777fe640 1 -- 172.21.15.18:0/671625493 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f1c84084c00 con 0x7f1c900af900 2024-03-29T11:11:34.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.226+0000 7f1c777fe640 10 client.4334.objecter ms_dispatch 0x7f1c90000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:11:34.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.226+0000 7f1c777fe640 3 client.4334.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:11:34.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.226+0000 7f1c777fe640 3 client.4334.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:11:34.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.227+0000 7f1c777fe640 20 client.4334.objecter dump_active .. 0 homeless 2024-03-29T11:11:34.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.227+0000 7f1c8effd640 10 monclient: get_auth_request con 0x7f1c7c06dd30 auth_method 0 2024-03-29T11:11:34.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.227+0000 7f1c777fe640 10 client.4334.objecter ms_handle_connect 0x7f1c7c06dd30 2024-03-29T11:11:34.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.234+0000 7f1c777fe640 1 -- 172.21.15.18:0/671625493 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1c84050540 con 0x7f1c900af900 2024-03-29T11:11:34.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.234+0000 7f1c777fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:34.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.234+0000 7f1c777fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:11:34.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.561+0000 7f1c954e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:34.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.561+0000 7f1c954e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:34.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.561+0000 7f1c954e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:11:34.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.561+0000 7f1c954e6640 1 -- 172.21.15.18:0/671625493 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c900ae080 con 0x7f1c900af900 2024-03-29T11:11:34.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.564+0000 7f1c777fe640 1 -- 172.21.15.18:0/671625493 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f1c840842b0 con 0x7f1c900af900 2024-03-29T11:11:34.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.564+0000 7f1c777fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:34.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.564+0000 7f1c777fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:11:34.543 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:11:34.543 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:11:34.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.567+0000 7f1c954e6640 10 librados: watch_flush enter 2024-03-29T11:11:34.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.567+0000 7f1c954e6640 10 librados: watch_flush exit 2024-03-29T11:11:34.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.568+0000 7f1c954e6640 20 client.4334.objecter shutdown clearing up homeless session... 2024-03-29T11:11:34.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.568+0000 7f1c954e6640 10 client.4334.objecter successfully canceled tick 2024-03-29T11:11:34.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.568+0000 7f1c954e6640 1 -- 172.21.15.18:0/671625493 >> v1:172.21.15.70:6816/3101871562 conn(0x7f1c7c06dd30 legacy=0x7f1c7c0701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:34.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.568+0000 7f1c954e6640 10 monclient: shutdown 2024-03-29T11:11:34.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.568+0000 7f1c954e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:34.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.568+0000 7f1c954e6640 1 -- 172.21.15.18:0/671625493 >> v1:172.21.15.18:6790/0 conn(0x7f1c900af900 legacy=0x7f1c901db910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:34.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.568+0000 7f1c954e6640 1 -- 172.21.15.18:0/671625493 shutdown_connections 2024-03-29T11:11:34.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.568+0000 7f1c954e6640 1 -- 172.21.15.18:0/671625493 >> 172.21.15.18:0/671625493 conn(0x7f1c900b35e0 msgr2=0x7f1c900ad0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:34.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.569+0000 7f1c954e6640 1 -- 172.21.15.18:0/671625493 shutdown_connections 2024-03-29T11:11:34.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.569+0000 7f1c954e6640 1 -- 172.21.15.18:0/671625493 wait complete. 2024-03-29T11:11:34.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:34.570+0000 7f1c954e6640 1 librados: shutdown 2024-03-29T11:11:34.566 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:11:41.373 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:11:41.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.542+0000 7f6eaf40f640 10 monclient: get_monmap_and_config 2024-03-29T11:11:41.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.542+0000 7f6eaf40f640 10 monclient: build_initial_monmap 2024-03-29T11:11:41.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.542+0000 7f6eaf40f640 10 monclient: monmap: 2024-03-29T11:11:41.521 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:11:41.521 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:41.521 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:11:41.543294+0000 2024-03-29T11:11:41.521 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:11:41.543294+0000 2024-03-29T11:11:41.521 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:41.521 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:41.521 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:11:41.521 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:11:41.521 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:11:41.521 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:41.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.543+0000 7f6eaf40f640 1 Processor -- start 2024-03-29T11:11:41.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.543+0000 7f6eaf40f640 1 -- start start 2024-03-29T11:11:41.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.543+0000 7f6eaf40f640 10 monclient: init 2024-03-29T11:11:41.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.545+0000 7f6eaf40f640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:41.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.545+0000 7f6eaf40f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:11:41.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.545+0000 7f6eaf40f640 10 monclient(hunting): picked mon.noname-b con 0x7f6ea80b0770 addr v1:172.21.15.18:6790/0 2024-03-29T11:11:41.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.545+0000 7f6eaf40f640 10 monclient(hunting): picked mon.noname-c con 0x7f6ea80ad850 addr v1:172.21.15.70:6789/0 2024-03-29T11:11:41.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.545+0000 7f6eaf40f640 10 monclient(hunting): picked mon.noname-a con 0x7f6ea813c1b0 addr v1:172.21.15.18:6789/0 2024-03-29T11:11:41.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.545+0000 7f6eaf40f640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea80ac8e0 con 0x7f6ea813c1b0 2024-03-29T11:11:41.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.545+0000 7f6eaf40f640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea80ad330 con 0x7f6ea80ad850 2024-03-29T11:11:41.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.545+0000 7f6eaf40f640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea80acfe0 con 0x7f6ea80b0770 2024-03-29T11:11:41.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.545+0000 7f6eaf40f640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:41.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.545+0000 7f6eaf40f640 10 monclient(hunting): authenticate will time out at 927.817347s 2024-03-29T11:11:41.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.546+0000 7f6ead1ab640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f6ea80b0770 0x7f6ea8072070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:37524/0 (socket says 172.21.15.18:37524) 2024-03-29T11:11:41.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.546+0000 7f6ead1ab640 1 -- 172.21.15.18:0/2123063370 learned_addr learned my addr 172.21.15.18:0/2123063370 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:11:41.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.546+0000 7f6e9ffff640 1 -- 172.21.15.18:0/2123063370 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 959873850 0 0) 0x7f6ea80acfe0 con 0x7f6ea80b0770 2024-03-29T11:11:41.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.546+0000 7f6e9ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:11:41.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.546+0000 7f6e9ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:41.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.546+0000 7f6e9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:41.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.546+0000 7f6e9ffff640 10 monclient(hunting): my global_id is 4337 2024-03-29T11:11:41.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.546+0000 7f6e9ffff640 1 -- 172.21.15.18:0/2123063370 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e94003520 con 0x7f6ea80b0770 2024-03-29T11:11:41.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.546+0000 7f6e9ffff640 1 -- 172.21.15.18:0/2123063370 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3149856753 0 0) 0x7f6ea80ad330 con 0x7f6ea80ad850 2024-03-29T11:11:41.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.546+0000 7f6e9ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:11:41.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.546+0000 7f6e9ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:41.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.546+0000 7f6e9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:41.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.546+0000 7f6e9ffff640 10 monclient(hunting): my global_id is 4321 2024-03-29T11:11:41.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.546+0000 7f6e9ffff640 1 -- 172.21.15.18:0/2123063370 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ea80acfe0 con 0x7f6ea80ad850 2024-03-29T11:11:41.532 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.546+0000 7f6e9ffff640 1 -- 172.21.15.18:0/2123063370 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1070926849 0 0) 0x7f6e94003520 con 0x7f6ea80b0770 2024-03-29T11:11:41.532 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.546+0000 7f6e9ffff640 1 -- 172.21.15.18:0/2123063370 >> v1:172.21.15.70:6789/0 conn(0x7f6ea80ad850 legacy=0x7f6ea80aaae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:41.532 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.546+0000 7f6e9ffff640 1 -- 172.21.15.18:0/2123063370 >> v1:172.21.15.18:6789/0 conn(0x7f6ea813c1b0 legacy=0x7f6ea814a760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:11:41.532 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.547+0000 7f6e9ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:11:41.532 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.547+0000 7f6e9ffff640 1 monclient: found mon.noname-b 2024-03-29T11:11:41.532 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.547+0000 7f6e9ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:41.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.547+0000 7f6e9ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:11:41.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.547+0000 7f6e9ffff640 1 -- 172.21.15.18:0/2123063370 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea80ab1f0 con 0x7f6ea80b0770 2024-03-29T11:11:41.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.547+0000 7f6e9ffff640 10 monclient: _finish_auth 0 2024-03-29T11:11:41.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.547+0000 7f6e9ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:11:41.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.548+0000 7f6e9ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:41.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.548+0000 7f6e9ffff640 1 -- 172.21.15.18:0/2123063370 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f6ea4001690 con 0x7f6ea80b0770 2024-03-29T11:11:41.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.548+0000 7f6eaf40f640 5 monclient: authenticate success, global_id 4337 2024-03-29T11:11:41.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.548+0000 7f6eaf40f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:41.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.548+0000 7f6e9ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:41.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.548+0000 7f6e9ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:11:41.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.548+0000 7f6e9ffff640 10 monclient: dump: 2024-03-29T11:11:41.534 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:11:41.535 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:41.535 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:41.535 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:41.535 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:41.535 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:41.535 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:11:41.536 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:11:41.536 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:11:41.536 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:41.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.548+0000 7f6eaf40f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:41.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.548+0000 7f6e9ffff640 1 -- 172.21.15.18:0/2123063370 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ea40018b0 con 0x7f6ea80b0770 2024-03-29T11:11:41.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.548+0000 7f6e9ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:41.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.548+0000 7f6e9ffff640 1 -- 172.21.15.18:0/2123063370 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f6ea4001c40 con 0x7f6ea80b0770 2024-03-29T11:11:41.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.548+0000 7f6eaf40f640 10 monclient: get_monmap_and_config success 2024-03-29T11:11:41.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.548+0000 7f6eaf40f640 4 set_mon_vals no callback set 2024-03-29T11:11:41.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.548+0000 7f6e9ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:41.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.548+0000 7f6e9ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:41.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.548+0000 7f6e9ffff640 10 monclient: dump: 2024-03-29T11:11:41.537 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:11:41.538 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:41.538 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:41.538 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:41.538 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:41.538 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:41.538 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:11:41.539 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:11:41.539 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:11:41.539 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:41.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.549+0000 7f6eaf40f640 10 monclient: shutdown 2024-03-29T11:11:41.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.549+0000 7f6eaf40f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:41.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.549+0000 7f6eaf40f640 1 -- 172.21.15.18:0/2123063370 >> v1:172.21.15.18:6790/0 conn(0x7f6ea80b0770 legacy=0x7f6ea8072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:41.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.549+0000 7f6eaf40f640 1 -- 172.21.15.18:0/2123063370 shutdown_connections 2024-03-29T11:11:41.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.549+0000 7f6eaf40f640 1 -- 172.21.15.18:0/2123063370 >> 172.21.15.18:0/2123063370 conn(0x7f6ea806b660 msgr2=0x7f6ea806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:41.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.549+0000 7f6eaf40f640 1 -- 172.21.15.18:0/2123063370 shutdown_connections 2024-03-29T11:11:41.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.549+0000 7f6eaf40f640 1 -- 172.21.15.18:0/2123063370 wait complete. 2024-03-29T11:11:41.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.550+0000 7f6eaf40f640 10 monclient: build_initial_monmap 2024-03-29T11:11:41.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.550+0000 7f6eaf40f640 10 monclient: monmap: 2024-03-29T11:11:41.555 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:11:41.555 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:11:41.555 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:11:41.555 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:11:41.555 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:41.556 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:41.556 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:11:41.556 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:11:41.556 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:11:41.556 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:41.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.551+0000 7f6eaf40f640 1 librados: starting msgr at 2024-03-29T11:11:41.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.551+0000 7f6eaf40f640 1 librados: starting objecter 2024-03-29T11:11:41.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.551+0000 7f6eaf40f640 1 Processor -- start 2024-03-29T11:11:41.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.551+0000 7f6eaf40f640 1 -- start start 2024-03-29T11:11:41.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.551+0000 7f6eaf40f640 1 librados: setting wanted keys 2024-03-29T11:11:41.557 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.551+0000 7f6eaf40f640 1 librados: calling monclient init 2024-03-29T11:11:41.557 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.551+0000 7f6eaf40f640 10 monclient: init 2024-03-29T11:11:41.557 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.552+0000 7f6eaf40f640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:41.557 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.552+0000 7f6eaf40f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:11:41.557 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.552+0000 7f6eaf40f640 10 monclient(hunting): picked mon.noname-b con 0x7f6ea80ad850 addr v1:172.21.15.70:6789/0 2024-03-29T11:11:41.557 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.552+0000 7f6ead1ab640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f6ea80ad850 0x7f6ea812e070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.18:33052/0 (socket says 172.21.15.18:33052) 2024-03-29T11:11:41.557 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.553+0000 7f6eaf40f640 10 monclient(hunting): picked mon.noname-a con 0x7f6ea80b0770 addr v1:172.21.15.18:6789/0 2024-03-29T11:11:41.557 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.553+0000 7f6ead1ab640 1 -- 172.21.15.18:0/3092560676 learned_addr learned my addr 172.21.15.18:0/3092560676 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:11:41.557 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.553+0000 7f6eaf40f640 10 monclient(hunting): picked mon.noname-c con 0x7f6ea813c1b0 addr v1:172.21.15.18:6790/0 2024-03-29T11:11:41.557 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.553+0000 7f6eaf40f640 1 -- 172.21.15.18:0/3092560676 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea812f5d0 con 0x7f6ea80b0770 2024-03-29T11:11:41.557 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.553+0000 7f6eaf40f640 1 -- 172.21.15.18:0/3092560676 --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea812e780 con 0x7f6ea80ad850 2024-03-29T11:11:41.557 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.554+0000 7f6e9dffb640 10 client.?.objecter ms_handle_connect 0x7f6ea80ad850 2024-03-29T11:11:41.557 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.554+0000 7f6e9dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:41.557 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.554+0000 7f6e9dffb640 10 client.?.objecter ms_handle_connect 0x7f6ea813c1b0 2024-03-29T11:11:41.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.554+0000 7f6e9dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:41.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.554+0000 7f6eaf40f640 1 -- 172.21.15.18:0/3092560676 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea81361d0 con 0x7f6ea813c1b0 2024-03-29T11:11:41.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.554+0000 7f6eaf40f640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:41.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.554+0000 7f6eaf40f640 10 monclient(hunting): authenticate will time out at 927.826187s 2024-03-29T11:11:41.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.554+0000 7f6e9dffb640 1 -- 172.21.15.18:0/3092560676 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4108050071 0 0) 0x7f6ea81361d0 con 0x7f6ea813c1b0 2024-03-29T11:11:41.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.554+0000 7f6e9dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:11:41.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.554+0000 7f6e9dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:41.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.554+0000 7f6e9dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:41.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.554+0000 7f6e9dffb640 10 monclient(hunting): my global_id is 4340 2024-03-29T11:11:41.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.554+0000 7f6e9dffb640 1 -- 172.21.15.18:0/3092560676 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e8c004eb0 con 0x7f6ea813c1b0 2024-03-29T11:11:41.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.554+0000 7f6e9dffb640 1 -- 172.21.15.18:0/3092560676 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3352386573 0 0) 0x7f6ea812e780 con 0x7f6ea80ad850 2024-03-29T11:11:41.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.554+0000 7f6e9dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:11:41.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.554+0000 7f6e9dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:41.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.554+0000 7f6e9dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:41.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.554+0000 7f6e9dffb640 10 monclient(hunting): my global_id is 4324 2024-03-29T11:11:41.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.554+0000 7f6e9dffb640 1 -- 172.21.15.18:0/3092560676 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ea81361d0 con 0x7f6ea80ad850 2024-03-29T11:11:41.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.554+0000 7f6e9dffb640 1 -- 172.21.15.18:0/3092560676 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2936382031 0 0) 0x7f6e8c004eb0 con 0x7f6ea813c1b0 2024-03-29T11:11:41.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.554+0000 7f6e9dffb640 1 -- 172.21.15.18:0/3092560676 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ea812e780 con 0x7f6ea813c1b0 2024-03-29T11:11:41.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.554+0000 7f6e9dffb640 1 -- 172.21.15.18:0/3092560676 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f6ea0001ad0 con 0x7f6ea813c1b0 2024-03-29T11:11:41.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.554+0000 7f6e9dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:41.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.554+0000 7f6e9dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:11:41.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.554+0000 7f6e9dffb640 10 monclient(hunting): dump: 2024-03-29T11:11:41.560 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:11:41.560 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:41.560 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:41.560 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:41.560 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:41.560 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:41.560 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:11:41.560 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:11:41.561 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:11:41.561 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:41.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.554+0000 7f6e9dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:11:41.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.554+0000 7f6e9dffb640 1 -- 172.21.15.18:0/3092560676 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3138854833 0 0) 0x7f6ea81361d0 con 0x7f6ea80ad850 2024-03-29T11:11:41.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.554+0000 7f6e9dffb640 1 -- 172.21.15.18:0/3092560676 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e8c004eb0 con 0x7f6ea80ad850 2024-03-29T11:11:41.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.554+0000 7f6e9dffb640 1 -- 172.21.15.18:0/3092560676 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f6ea4001610 con 0x7f6ea80ad850 2024-03-29T11:11:41.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.554+0000 7f6e9dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:41.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.554+0000 7f6e9dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:11:41.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.554+0000 7f6e9dffb640 10 monclient(hunting): dump: 2024-03-29T11:11:41.561 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:11:41.561 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:41.562 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:41.562 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:41.562 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:41.562 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:41.562 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:11:41.562 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:11:41.562 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:11:41.562 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:41.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.554+0000 7f6e9dffb640 1 -- 172.21.15.18:0/3092560676 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2104656687 0 0) 0x7f6ea812e780 con 0x7f6ea813c1b0 2024-03-29T11:11:41.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.554+0000 7f6e9dffb640 1 -- 172.21.15.18:0/3092560676 >> v1:172.21.15.70:6789/0 conn(0x7f6ea80ad850 legacy=0x7f6ea812e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:41.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.554+0000 7f6e9dffb640 1 -- 172.21.15.18:0/3092560676 >> v1:172.21.15.18:6789/0 conn(0x7f6ea80b0770 legacy=0x7f6ea81dce20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:11:41.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.555+0000 7f6e9dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:11:41.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.555+0000 7f6e9dffb640 1 monclient: found mon.c 2024-03-29T11:11:41.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.555+0000 7f6e9dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:41.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.555+0000 7f6e9dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:11:41.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.555+0000 7f6e9dffb640 1 -- 172.21.15.18:0/3092560676 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea812d4f0 con 0x7f6ea813c1b0 2024-03-29T11:11:41.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.555+0000 7f6e9dffb640 10 monclient: _finish_auth 0 2024-03-29T11:11:41.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.555+0000 7f6e9dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:11:41.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.555+0000 7f6e9dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:41.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.555+0000 7f6eaf40f640 5 monclient: authenticate success, global_id 4340 2024-03-29T11:11:41.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.555+0000 7f6eaf40f640 10 monclient: _renew_subs 2024-03-29T11:11:41.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.555+0000 7f6eaf40f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:11:41.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.555+0000 7f6eaf40f640 1 -- 172.21.15.18:0/3092560676 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ea8071b30 con 0x7f6ea813c1b0 2024-03-29T11:11:41.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.555+0000 7f6eaf40f640 10 client.4340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:11:41.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.555+0000 7f6eaf40f640 10 monclient: _renew_subs 2024-03-29T11:11:41.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.555+0000 7f6eaf40f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:11:41.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.555+0000 7f6eaf40f640 1 -- 172.21.15.18:0/3092560676 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ea812dc60 con 0x7f6ea813c1b0 2024-03-29T11:11:41.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.555+0000 7f6eaf40f640 1 librados: init done 2024-03-29T11:11:41.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.557+0000 7f6e9dffb640 1 -- 172.21.15.18:0/3092560676 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ea0001fe0 con 0x7f6ea813c1b0 2024-03-29T11:11:41.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.557+0000 7f6e9dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:41.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.557+0000 7f6e9dffb640 1 -- 172.21.15.18:0/3092560676 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f6ea0003b10 con 0x7f6ea813c1b0 2024-03-29T11:11:41.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.557+0000 7f6e9dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:41.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.557+0000 7f6e9dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:41.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.557+0000 7f6e9dffb640 10 monclient: dump: 2024-03-29T11:11:41.565 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:11:41.565 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:41.565 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:41.565 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:41.565 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:41.566 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:41.566 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:11:41.566 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:11:41.566 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:11:41.566 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:41.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.557+0000 7f6e9f7fe640 4 set_mon_vals no callback set 2024-03-29T11:11:41.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.557+0000 7f6e9dffb640 1 -- 172.21.15.18:0/3092560676 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f6ea0019ae0 con 0x7f6ea813c1b0 2024-03-29T11:11:41.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.558+0000 7f6eaf40f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:41.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.558+0000 7f6eaf40f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:41.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.558+0000 7f6eaf40f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:11:41.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.558+0000 7f6eaf40f640 1 -- 172.21.15.18:0/3092560676 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e88005080 con 0x7f6ea813c1b0 2024-03-29T11:11:41.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.558+0000 7f6e9dffb640 1 -- 172.21.15.18:0/3092560676 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f6ea0083a70 con 0x7f6ea813c1b0 2024-03-29T11:11:41.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.558+0000 7f6e9dffb640 10 client.4340.objecter ms_dispatch 0x7f6ea8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:11:41.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.558+0000 7f6e9dffb640 3 client.4340.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:11:41.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.558+0000 7f6e9dffb640 3 client.4340.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:11:41.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.558+0000 7f6e9dffb640 20 client.4340.objecter dump_active .. 0 homeless 2024-03-29T11:11:41.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.558+0000 7f6ead1ab640 10 monclient: get_auth_request con 0x7f6e8c076740 auth_method 0 2024-03-29T11:11:41.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.558+0000 7f6e9dffb640 10 client.4340.objecter ms_handle_connect 0x7f6e8c076740 2024-03-29T11:11:41.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.566+0000 7f6e9dffb640 1 -- 172.21.15.18:0/3092560676 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6ea004f480 con 0x7f6ea813c1b0 2024-03-29T11:11:41.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.566+0000 7f6e9dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:41.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.566+0000 7f6e9dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:11:41.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.876+0000 7f6eaf40f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:41.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.876+0000 7f6eaf40f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:41.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.876+0000 7f6eaf40f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:11:41.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.876+0000 7f6eaf40f640 1 -- 172.21.15.18:0/3092560676 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e88005330 con 0x7f6ea813c1b0 2024-03-29T11:11:41.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.877+0000 7f6e9dffb640 1 -- 172.21.15.18:0/3092560676 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f6ea00513b0 con 0x7f6ea813c1b0 2024-03-29T11:11:41.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.877+0000 7f6e9dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:41.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.877+0000 7f6e9dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:11:41.857 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:11:41.857 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:11:41.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.880+0000 7f6eaf40f640 10 librados: watch_flush enter 2024-03-29T11:11:41.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.880+0000 7f6eaf40f640 10 librados: watch_flush exit 2024-03-29T11:11:41.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.880+0000 7f6eaf40f640 20 client.4340.objecter shutdown clearing up homeless session... 2024-03-29T11:11:41.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.880+0000 7f6eaf40f640 10 client.4340.objecter successfully canceled tick 2024-03-29T11:11:41.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.880+0000 7f6eaf40f640 1 -- 172.21.15.18:0/3092560676 >> v1:172.21.15.70:6816/3101871562 conn(0x7f6e8c076740 legacy=0x7f6e8c078c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:41.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.880+0000 7f6eaf40f640 10 monclient: shutdown 2024-03-29T11:11:41.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.880+0000 7f6eaf40f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:41.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.880+0000 7f6eaf40f640 1 -- 172.21.15.18:0/3092560676 >> v1:172.21.15.18:6790/0 conn(0x7f6ea813c1b0 legacy=0x7f6ea81e0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:41.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.881+0000 7f6eaf40f640 1 -- 172.21.15.18:0/3092560676 shutdown_connections 2024-03-29T11:11:41.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.881+0000 7f6eaf40f640 1 -- 172.21.15.18:0/3092560676 >> 172.21.15.18:0/3092560676 conn(0x7f6ea806b660 msgr2=0x7f6ea80b3d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:41.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.881+0000 7f6eaf40f640 1 -- 172.21.15.18:0/3092560676 shutdown_connections 2024-03-29T11:11:41.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.881+0000 7f6eaf40f640 1 -- 172.21.15.18:0/3092560676 wait complete. 2024-03-29T11:11:41.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:41.881+0000 7f6eaf40f640 1 librados: shutdown 2024-03-29T11:11:41.875 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:11:48.683 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:11:48.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.848+0000 7f267e8d0640 10 monclient: get_monmap_and_config 2024-03-29T11:11:48.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.848+0000 7f267e8d0640 10 monclient: build_initial_monmap 2024-03-29T11:11:48.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.848+0000 7f267e8d0640 10 monclient: monmap: 2024-03-29T11:11:48.826 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:11:48.827 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:48.827 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:11:48.849117+0000 2024-03-29T11:11:48.827 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:11:48.849117+0000 2024-03-29T11:11:48.827 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:48.827 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:48.827 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:11:48.827 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:11:48.827 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:11:48.827 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:48.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.849+0000 7f267e8d0640 1 Processor -- start 2024-03-29T11:11:48.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.849+0000 7f267e8d0640 1 -- start start 2024-03-29T11:11:48.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.849+0000 7f267e8d0640 10 monclient: init 2024-03-29T11:11:48.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.850+0000 7f267e8d0640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:48.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.850+0000 7f267e8d0640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:11:48.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.850+0000 7f267e8d0640 10 monclient(hunting): picked mon.noname-b con 0x7f2670006890 addr v1:172.21.15.18:6790/0 2024-03-29T11:11:48.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.853+0000 7f267e8d0640 10 monclient(hunting): picked mon.noname-a con 0x7f2670006c70 addr v1:172.21.15.18:6789/0 2024-03-29T11:11:48.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.853+0000 7f267d0cd640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f2670006c70 0x7f26700c6d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:37911/0 (socket says 172.21.15.18:37911) 2024-03-29T11:11:48.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.853+0000 7f267d8ce640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f2670006890 0x7f26700c0a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:53016/0 (socket says 172.21.15.18:53016) 2024-03-29T11:11:48.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.853+0000 7f267e8d0640 10 monclient(hunting): picked mon.noname-c con 0x7f267002b4c0 addr v1:172.21.15.70:6789/0 2024-03-29T11:11:48.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.853+0000 7f267e8d0640 1 -- 172.21.15.18:0/2634743824 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f267002c830 con 0x7f2670006c70 2024-03-29T11:11:48.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.853+0000 7f267e8d0640 1 -- 172.21.15.18:0/2634743824 --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f267002afa0 con 0x7f267002b4c0 2024-03-29T11:11:48.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.853+0000 7f267e8d0640 1 -- 172.21.15.18:0/2634743824 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f267002aba0 con 0x7f2670006890 2024-03-29T11:11:48.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.853+0000 7f267e8d0640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:48.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.853+0000 7f267e8d0640 10 monclient(hunting): authenticate will time out at 935.125754s 2024-03-29T11:11:48.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.853+0000 7f267d0cd640 1 -- 172.21.15.18:0/2634743824 learned_addr learned my addr 172.21.15.18:0/2634743824 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:11:48.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.854+0000 7f267c8cc640 1 -- 172.21.15.18:0/2634743824 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2252273756 0 0) 0x7f267002c830 con 0x7f2670006c70 2024-03-29T11:11:48.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.854+0000 7f267c8cc640 10 monclient(hunting): _negotiate 2024-03-29T11:11:48.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.854+0000 7f267c8cc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:48.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.854+0000 7f267c8cc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:48.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.854+0000 7f267c8cc640 10 monclient(hunting): my global_id is 4317 2024-03-29T11:11:48.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.854+0000 7f267c8cc640 1 -- 172.21.15.18:0/2634743824 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2664003520 con 0x7f2670006c70 2024-03-29T11:11:48.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.854+0000 7f267c8cc640 1 -- 172.21.15.18:0/2634743824 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1827364447 0 0) 0x7f267002aba0 con 0x7f2670006890 2024-03-29T11:11:48.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.854+0000 7f267c8cc640 10 monclient(hunting): _negotiate 2024-03-29T11:11:48.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.854+0000 7f267c8cc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:48.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.854+0000 7f267c8cc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:48.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.854+0000 7f267c8cc640 10 monclient(hunting): my global_id is 4343 2024-03-29T11:11:48.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.854+0000 7f267c8cc640 1 -- 172.21.15.18:0/2634743824 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f267002c830 con 0x7f2670006890 2024-03-29T11:11:48.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.855+0000 7f267c8cc640 1 -- 172.21.15.18:0/2634743824 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3135619532 0 0) 0x7f2664003520 con 0x7f2670006c70 2024-03-29T11:11:48.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.855+0000 7f267c8cc640 1 -- 172.21.15.18:0/2634743824 >> v1:172.21.15.18:6790/0 conn(0x7f2670006890 legacy=0x7f26700c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:48.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.855+0000 7f267c8cc640 1 -- 172.21.15.18:0/2634743824 >> v1:172.21.15.70:6789/0 conn(0x7f267002b4c0 legacy=0x7f26700cf0e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:11:48.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.856+0000 7f267c8cc640 10 monclient: _finish_hunting 0 2024-03-29T11:11:48.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.856+0000 7f267c8cc640 1 monclient: found mon.noname-a 2024-03-29T11:11:48.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.856+0000 7f267c8cc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:48.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.856+0000 7f267c8cc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-29T11:11:48.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.856+0000 7f267c8cc640 1 -- 172.21.15.18:0/2634743824 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f267002ad30 con 0x7f2670006c70 2024-03-29T11:11:48.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.857+0000 7f267c8cc640 10 monclient: _finish_auth 0 2024-03-29T11:11:48.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.857+0000 7f267c8cc640 10 monclient: _check_auth_tickets 2024-03-29T11:11:48.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.857+0000 7f267c8cc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:48.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.857+0000 7f267e8d0640 5 monclient: authenticate success, global_id 4317 2024-03-29T11:11:48.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.857+0000 7f267e8d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:48.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.857+0000 7f267c8cc640 1 -- 172.21.15.18:0/2634743824 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f2674001690 con 0x7f2670006c70 2024-03-29T11:11:48.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.857+0000 7f267c8cc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:48.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.857+0000 7f267c8cc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:11:48.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.857+0000 7f267c8cc640 10 monclient: dump: 2024-03-29T11:11:48.838 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:11:48.838 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:48.838 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:48.838 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:48.838 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:48.838 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:48.839 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:11:48.839 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:11:48.839 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:11:48.839 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:48.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.857+0000 7f267e8d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:48.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.857+0000 7f267c8cc640 1 -- 172.21.15.18:0/2634743824 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26740018b0 con 0x7f2670006c70 2024-03-29T11:11:48.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.857+0000 7f267c8cc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:48.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.857+0000 7f267c8cc640 1 -- 172.21.15.18:0/2634743824 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f2674001bc0 con 0x7f2670006c70 2024-03-29T11:11:48.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.857+0000 7f267c8cc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:48.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.858+0000 7f267c8cc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:11:48.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.858+0000 7f267c8cc640 10 monclient: dump: 2024-03-29T11:11:48.840 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:11:48.840 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:48.840 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:48.840 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:48.840 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:48.840 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:48.841 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:11:48.841 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:11:48.841 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:11:48.841 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:48.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.858+0000 7f267e8d0640 10 monclient: get_monmap_and_config success 2024-03-29T11:11:48.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.858+0000 7f267e8d0640 4 set_mon_vals no callback set 2024-03-29T11:11:48.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.859+0000 7f267e8d0640 10 monclient: shutdown 2024-03-29T11:11:48.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.859+0000 7f267e8d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:48.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.859+0000 7f267e8d0640 1 -- 172.21.15.18:0/2634743824 >> v1:172.21.15.18:6789/0 conn(0x7f2670006c70 legacy=0x7f26700c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:48.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.860+0000 7f267e8d0640 1 -- 172.21.15.18:0/2634743824 shutdown_connections 2024-03-29T11:11:48.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.860+0000 7f267e8d0640 1 -- 172.21.15.18:0/2634743824 >> 172.21.15.18:0/2634743824 conn(0x7f26700058f0 msgr2=0x7f2670005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:48.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.861+0000 7f267e8d0640 1 -- 172.21.15.18:0/2634743824 shutdown_connections 2024-03-29T11:11:48.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.861+0000 7f267e8d0640 1 -- 172.21.15.18:0/2634743824 wait complete. 2024-03-29T11:11:48.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.862+0000 7f267e8d0640 10 monclient: build_initial_monmap 2024-03-29T11:11:48.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.862+0000 7f267e8d0640 10 monclient: monmap: 2024-03-29T11:11:48.844 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:11:48.844 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:11:48.844 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:11:48.844 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:11:48.844 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:48.844 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:48.844 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:11:48.844 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:11:48.844 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:11:48.845 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:48.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.863+0000 7f267e8d0640 1 librados: starting msgr at 2024-03-29T11:11:48.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.863+0000 7f267e8d0640 1 librados: starting objecter 2024-03-29T11:11:48.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.863+0000 7f267e8d0640 1 Processor -- start 2024-03-29T11:11:48.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.863+0000 7f267e8d0640 1 -- start start 2024-03-29T11:11:48.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.864+0000 7f267e8d0640 1 librados: setting wanted keys 2024-03-29T11:11:48.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.864+0000 7f267e8d0640 1 librados: calling monclient init 2024-03-29T11:11:48.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.864+0000 7f267e8d0640 10 monclient: init 2024-03-29T11:11:48.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.865+0000 7f267e8d0640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:48.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.865+0000 7f267e8d0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:11:48.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.865+0000 7f267e8d0640 10 monclient(hunting): picked mon.noname-c con 0x7f2670006890 addr v1:172.21.15.18:6790/0 2024-03-29T11:11:48.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.865+0000 7f267d8ce640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f2670006890 0x7f26700c03f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:53532/0 (socket says 172.21.15.18:53532) 2024-03-29T11:11:48.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.867+0000 7f267d8ce640 1 -- 172.21.15.18:0/1601285505 learned_addr learned my addr 172.21.15.18:0/1601285505 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:11:48.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.867+0000 7f267e8d0640 10 monclient(hunting): picked mon.noname-b con 0x7f267002b4c0 addr v1:172.21.15.70:6789/0 2024-03-29T11:11:48.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.867+0000 7f267e8d0640 10 monclient(hunting): picked mon.noname-a con 0x7f26700bb540 addr v1:172.21.15.18:6789/0 2024-03-29T11:11:48.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.867+0000 7f267e8d0640 1 -- 172.21.15.18:0/1601285505 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2670031840 con 0x7f26700bb540 2024-03-29T11:11:48.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.867+0000 7f267e8d0640 1 -- 172.21.15.18:0/1601285505 --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2670031b20 con 0x7f267002b4c0 2024-03-29T11:11:48.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.867+0000 7f267e8d0640 1 -- 172.21.15.18:0/1601285505 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2670156bc0 con 0x7f2670006890 2024-03-29T11:11:48.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.867+0000 7f267e8d0640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:48.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.867+0000 7f267e8d0640 10 monclient(hunting): authenticate will time out at 935.139520s 2024-03-29T11:11:48.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.867+0000 7f266e7fc640 10 client.?.objecter ms_handle_connect 0x7f2670006890 2024-03-29T11:11:48.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.867+0000 7f266e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:48.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.867+0000 7f266e7fc640 1 -- 172.21.15.18:0/1601285505 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3352379301 0 0) 0x7f2670156bc0 con 0x7f2670006890 2024-03-29T11:11:48.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.867+0000 7f266e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:11:48.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.867+0000 7f266e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:48.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.867+0000 7f266e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:48.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.867+0000 7f266e7fc640 10 monclient(hunting): my global_id is 4346 2024-03-29T11:11:48.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.867+0000 7f266e7fc640 1 -- 172.21.15.18:0/1601285505 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f265c0070c0 con 0x7f2670006890 2024-03-29T11:11:48.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.868+0000 7f266e7fc640 1 -- 172.21.15.18:0/1601285505 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 16179931 0 0) 0x7f265c0070c0 con 0x7f2670006890 2024-03-29T11:11:48.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.868+0000 7f266e7fc640 1 -- 172.21.15.18:0/1601285505 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2670156bc0 con 0x7f2670006890 2024-03-29T11:11:48.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.868+0000 7f266e7fc640 1 -- 172.21.15.18:0/1601285505 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f26680017e0 con 0x7f2670006890 2024-03-29T11:11:48.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.868+0000 7f266e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:48.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.868+0000 7f266e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:11:48.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.868+0000 7f266e7fc640 10 monclient(hunting): dump: 2024-03-29T11:11:48.850 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:11:48.850 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:48.850 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:48.850 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:48.850 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:48.850 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:48.850 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:11:48.850 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:11:48.850 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:11:48.850 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:48.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.868+0000 7f266e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:11:48.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.868+0000 7f266e7fc640 1 -- 172.21.15.18:0/1601285505 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2202857108 0 0) 0x7f2670156bc0 con 0x7f2670006890 2024-03-29T11:11:48.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.868+0000 7f266e7fc640 1 -- 172.21.15.18:0/1601285505 >> v1:172.21.15.70:6789/0 conn(0x7f267002b4c0 legacy=0x7f2670155da0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:11:48.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.869+0000 7f266e7fc640 1 -- 172.21.15.18:0/1601285505 >> v1:172.21.15.18:6789/0 conn(0x7f26700bb540 legacy=0x7f26701564b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:11:48.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.872+0000 7f266e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:11:48.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.872+0000 7f266e7fc640 1 monclient: found mon.c 2024-03-29T11:11:48.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.872+0000 7f266e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:48.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.872+0000 7f266e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:11:48.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.872+0000 7f266e7fc640 1 -- 172.21.15.18:0/1601285505 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26700bad40 con 0x7f2670006890 2024-03-29T11:11:48.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.872+0000 7f266e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:11:48.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.872+0000 7f266e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:11:48.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.872+0000 7f266e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:48.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.872+0000 7f267e8d0640 5 monclient: authenticate success, global_id 4346 2024-03-29T11:11:48.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.872+0000 7f267e8d0640 10 monclient: _renew_subs 2024-03-29T11:11:48.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.872+0000 7f267e8d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:11:48.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.872+0000 7f267e8d0640 1 -- 172.21.15.18:0/1601285505 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26700bba00 con 0x7f2670006890 2024-03-29T11:11:48.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.872+0000 7f267e8d0640 10 client.4346.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:11:48.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.872+0000 7f267e8d0640 10 monclient: _renew_subs 2024-03-29T11:11:48.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.873+0000 7f267e8d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:11:48.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.873+0000 7f267e8d0640 1 -- 172.21.15.18:0/1601285505 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f26700c0ce0 con 0x7f2670006890 2024-03-29T11:11:48.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.873+0000 7f267e8d0640 1 librados: init done 2024-03-29T11:11:48.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.874+0000 7f267e8d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:48.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.874+0000 7f267e8d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:48.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.874+0000 7f267e8d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:11:48.855 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.874+0000 7f267e8d0640 1 -- 172.21.15.18:0/1601285505 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2670031cd0 con 0x7f2670006890 2024-03-29T11:11:48.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.876+0000 7f266e7fc640 1 -- 172.21.15.18:0/1601285505 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2668001940 con 0x7f2670006890 2024-03-29T11:11:48.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.876+0000 7f266e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:48.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.876+0000 7f266e7fc640 1 -- 172.21.15.18:0/1601285505 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f26680049c0 con 0x7f2670006890 2024-03-29T11:11:48.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.876+0000 7f266e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:48.856 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.876+0000 7f266e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:48.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.876+0000 7f266e7fc640 10 monclient: dump: 2024-03-29T11:11:48.857 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:11:48.857 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:48.857 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:48.857 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:48.857 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:48.857 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:48.857 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:11:48.857 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:11:48.857 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:11:48.857 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:48.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.877+0000 7f266e7fc640 1 -- 172.21.15.18:0/1601285505 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f266801aa50 con 0x7f2670006890 2024-03-29T11:11:48.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.877+0000 7f266e7fc640 1 -- 172.21.15.18:0/1601285505 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f2668084ab0 con 0x7f2670006890 2024-03-29T11:11:48.857 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.877+0000 7f266e7fc640 10 client.4346.objecter ms_dispatch 0x7f2678000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:11:48.858 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.877+0000 7f266e7fc640 3 client.4346.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:11:48.858 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.877+0000 7f266e7fc640 3 client.4346.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:11:48.858 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.878+0000 7f266e7fc640 20 client.4346.objecter dump_active .. 0 homeless 2024-03-29T11:11:48.858 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.878+0000 7f266ffff640 4 set_mon_vals no callback set 2024-03-29T11:11:48.858 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.879+0000 7f267d0cd640 10 monclient: get_auth_request con 0x7f265c072110 auth_method 0 2024-03-29T11:11:48.858 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.879+0000 7f266e7fc640 10 client.4346.objecter ms_handle_connect 0x7f265c072110 2024-03-29T11:11:48.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.884+0000 7f266e7fc640 1 -- 172.21.15.18:0/1601285505 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2670031cd0 con 0x7f2670006890 2024-03-29T11:11:48.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.884+0000 7f266e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:48.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:48.884+0000 7f266e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:11:49.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:49.214+0000 7f267e8d0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:49.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:49.214+0000 7f267e8d0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:49.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:49.214+0000 7f267e8d0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:11:49.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:49.214+0000 7f267e8d0640 1 -- 172.21.15.18:0/1601285505 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2670031cd0 con 0x7f2670006890 2024-03-29T11:11:49.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:49.218+0000 7f266e7fc640 1 -- 172.21.15.18:0/1601285505 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f2670031cd0 con 0x7f2670006890 2024-03-29T11:11:49.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:49.218+0000 7f266e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:49.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:49.218+0000 7f266e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:11:49.196 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:11:49.196 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:11:49.200 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:49.221+0000 7f267e8d0640 10 librados: watch_flush enter 2024-03-29T11:11:49.200 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:49.221+0000 7f267e8d0640 10 librados: watch_flush exit 2024-03-29T11:11:49.200 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:49.221+0000 7f267e8d0640 20 client.4346.objecter shutdown clearing up homeless session... 2024-03-29T11:11:49.200 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:49.221+0000 7f267e8d0640 10 client.4346.objecter successfully canceled tick 2024-03-29T11:11:49.200 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:49.221+0000 7f267e8d0640 1 -- 172.21.15.18:0/1601285505 >> v1:172.21.15.70:6816/3101871562 conn(0x7f265c072110 legacy=0x7f265c0745d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:49.200 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:49.221+0000 7f267e8d0640 10 monclient: shutdown 2024-03-29T11:11:49.200 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:49.222+0000 7f267e8d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:49.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:49.222+0000 7f267e8d0640 1 -- 172.21.15.18:0/1601285505 >> v1:172.21.15.18:6790/0 conn(0x7f2670006890 legacy=0x7f26700c03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:49.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:49.222+0000 7f267e8d0640 1 -- 172.21.15.18:0/1601285505 shutdown_connections 2024-03-29T11:11:49.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:49.222+0000 7f267e8d0640 1 -- 172.21.15.18:0/1601285505 >> 172.21.15.18:0/1601285505 conn(0x7f26700058f0 msgr2=0x7f2670038130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:49.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:49.223+0000 7f267e8d0640 1 -- 172.21.15.18:0/1601285505 shutdown_connections 2024-03-29T11:11:49.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:49.223+0000 7f267e8d0640 1 -- 172.21.15.18:0/1601285505 wait complete. 2024-03-29T11:11:49.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:49.223+0000 7f267e8d0640 1 librados: shutdown 2024-03-29T11:11:49.220 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:11:56.029 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:11:56.180 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.202+0000 7f2be176d640 10 monclient: get_monmap_and_config 2024-03-29T11:11:56.180 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.202+0000 7f2be176d640 10 monclient: build_initial_monmap 2024-03-29T11:11:56.180 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.202+0000 7f2be176d640 10 monclient: monmap: 2024-03-29T11:11:56.180 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:11:56.180 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:56.180 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:11:56.203256+0000 2024-03-29T11:11:56.181 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:11:56.203256+0000 2024-03-29T11:11:56.181 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:56.181 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:56.181 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:11:56.181 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:11:56.181 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:11:56.181 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:56.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.203+0000 7f2be176d640 1 Processor -- start 2024-03-29T11:11:56.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.203+0000 7f2be176d640 1 -- start start 2024-03-29T11:11:56.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.203+0000 7f2be176d640 10 monclient: init 2024-03-29T11:11:56.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.204+0000 7f2be176d640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:56.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.204+0000 7f2be176d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:11:56.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.204+0000 7f2be176d640 10 monclient(hunting): picked mon.noname-a con 0x7f2bdc0b1f60 addr v1:172.21.15.18:6789/0 2024-03-29T11:11:56.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.204+0000 7f2be176d640 10 monclient(hunting): picked mon.noname-b con 0x7f2bdc071eb0 addr v1:172.21.15.18:6790/0 2024-03-29T11:11:56.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.204+0000 7f2be176d640 10 monclient(hunting): picked mon.noname-c con 0x7f2bdc143d10 addr v1:172.21.15.70:6789/0 2024-03-29T11:11:56.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.204+0000 7f2be176d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bdc06ef80 con 0x7f2bdc0b1f60 2024-03-29T11:11:56.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.204+0000 7f2be176d640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bdc06f260 con 0x7f2bdc143d10 2024-03-29T11:11:56.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.204+0000 7f2be176d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bdc0b40c0 con 0x7f2bdc071eb0 2024-03-29T11:11:56.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.204+0000 7f2be176d640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:56.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.204+0000 7f2be176d640 10 monclient(hunting): authenticate will time out at 942.476189s 2024-03-29T11:11:56.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.204+0000 7f2bdb7fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f2bdc143d10 0x7f2bdc146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.18:43822/0 (socket says 172.21.15.18:43822) 2024-03-29T11:11:56.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.204+0000 7f2bdb7fe640 1 -- 172.21.15.18:0/3150006210 learned_addr learned my addr 172.21.15.18:0/3150006210 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:11:56.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.205+0000 7f2bd9ffb640 1 -- 172.21.15.18:0/3150006210 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2433418289 0 0) 0x7f2bdc06f260 con 0x7f2bdc143d10 2024-03-29T11:11:56.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.205+0000 7f2bd9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:11:56.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.205+0000 7f2bd9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:56.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.205+0000 7f2bd9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:56.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.205+0000 7f2bd9ffb640 10 monclient(hunting): my global_id is 4327 2024-03-29T11:11:56.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.205+0000 7f2bd9ffb640 1 -- 172.21.15.18:0/3150006210 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bc4003520 con 0x7f2bdc143d10 2024-03-29T11:11:56.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.207+0000 7f2bd9ffb640 1 -- 172.21.15.18:0/3150006210 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3602589606 0 0) 0x7f2bc4003520 con 0x7f2bdc143d10 2024-03-29T11:11:56.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.207+0000 7f2bd9ffb640 1 -- 172.21.15.18:0/3150006210 >> v1:172.21.15.18:6790/0 conn(0x7f2bdc071eb0 legacy=0x7f2bdc146470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:11:56.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.208+0000 7f2bd9ffb640 1 -- 172.21.15.18:0/3150006210 >> v1:172.21.15.18:6789/0 conn(0x7f2bdc0b1f60 legacy=0x7f2bdc143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:56.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.208+0000 7f2bd9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:11:56.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.208+0000 7f2bd9ffb640 1 monclient: found mon.noname-c 2024-03-29T11:11:56.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.208+0000 7f2bd9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:56.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.208+0000 7f2bd9ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.70:6789/0 2024-03-29T11:11:56.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.208+0000 7f2bd9ffb640 1 -- 172.21.15.18:0/3150006210 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bdc155d70 con 0x7f2bdc143d10 2024-03-29T11:11:56.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.208+0000 7f2bd9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:11:56.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.208+0000 7f2bd9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:11:56.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.208+0000 7f2bd9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:56.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.208+0000 7f2bd9ffb640 1 -- 172.21.15.18:0/3150006210 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f2bd4001a10 con 0x7f2bdc143d10 2024-03-29T11:11:56.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.208+0000 7f2bd9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:56.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.208+0000 7f2bd9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:11:56.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.208+0000 7f2bd9ffb640 10 monclient: dump: 2024-03-29T11:11:56.189 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:11:56.189 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:56.190 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:56.190 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:56.190 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:56.190 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:56.190 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:11:56.190 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:11:56.190 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:11:56.190 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:56.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.208+0000 7f2be176d640 5 monclient: authenticate success, global_id 4327 2024-03-29T11:11:56.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.208+0000 7f2be176d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:11:56.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.209+0000 7f2bd9ffb640 1 -- 172.21.15.18:0/3150006210 <== mon.1 v1:172.21.15.70:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2bd4001be0 con 0x7f2bdc143d10 2024-03-29T11:11:56.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.209+0000 7f2bd9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:56.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.209+0000 7f2bd9ffb640 1 -- 172.21.15.18:0/3150006210 <== mon.1 v1:172.21.15.70:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f2bd4001ff0 con 0x7f2bdc143d10 2024-03-29T11:11:56.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.209+0000 7f2bd9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:56.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.209+0000 7f2bd9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:11:56.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.209+0000 7f2bd9ffb640 10 monclient: dump: 2024-03-29T11:11:56.191 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:11:56.191 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:56.191 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:56.191 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:56.191 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:56.191 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:56.191 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:11:56.192 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:11:56.192 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:11:56.192 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:56.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.209+0000 7f2be176d640 10 monclient: get_monmap_and_config success 2024-03-29T11:11:56.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.209+0000 7f2be176d640 4 set_mon_vals no callback set 2024-03-29T11:11:56.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.210+0000 7f2be176d640 10 monclient: shutdown 2024-03-29T11:11:56.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.210+0000 7f2be176d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:56.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.210+0000 7f2be176d640 1 -- 172.21.15.18:0/3150006210 >> v1:172.21.15.70:6789/0 conn(0x7f2bdc143d10 legacy=0x7f2bdc146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:56.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.210+0000 7f2be176d640 1 -- 172.21.15.18:0/3150006210 shutdown_connections 2024-03-29T11:11:56.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.210+0000 7f2be176d640 1 -- 172.21.15.18:0/3150006210 >> 172.21.15.18:0/3150006210 conn(0x7f2bdc066a00 msgr2=0x7f2bdc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:56.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.210+0000 7f2be176d640 1 -- 172.21.15.18:0/3150006210 shutdown_connections 2024-03-29T11:11:56.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.210+0000 7f2be176d640 1 -- 172.21.15.18:0/3150006210 wait complete. 2024-03-29T11:11:56.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.212+0000 7f2be176d640 10 monclient: build_initial_monmap 2024-03-29T11:11:56.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.212+0000 7f2be176d640 10 monclient: monmap: 2024-03-29T11:11:56.194 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:11:56.194 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:11:56.194 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:11:56.194 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:11:56.194 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:11:56.195 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:56.195 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:11:56.195 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:11:56.195 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:11:56.195 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:56.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.213+0000 7f2be176d640 1 librados: starting msgr at 2024-03-29T11:11:56.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.213+0000 7f2be176d640 1 librados: starting objecter 2024-03-29T11:11:56.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.214+0000 7f2be176d640 1 Processor -- start 2024-03-29T11:11:56.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.214+0000 7f2be176d640 1 -- start start 2024-03-29T11:11:56.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.214+0000 7f2be176d640 1 librados: setting wanted keys 2024-03-29T11:11:56.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.214+0000 7f2be176d640 1 librados: calling monclient init 2024-03-29T11:11:56.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.214+0000 7f2be176d640 10 monclient: init 2024-03-29T11:11:56.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.214+0000 7f2be176d640 10 monclient: _reopen_session rank -1 2024-03-29T11:11:56.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.214+0000 7f2be176d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:11:56.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.215+0000 7f2be176d640 10 monclient(hunting): picked mon.noname-a con 0x7f2bdc0b1f60 addr v1:172.21.15.18:6789/0 2024-03-29T11:11:56.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.215+0000 7f2be176d640 10 monclient(hunting): picked mon.noname-b con 0x7f2bdc143d10 addr v1:172.21.15.70:6789/0 2024-03-29T11:11:56.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.215+0000 7f2be176d640 10 monclient(hunting): picked mon.noname-c con 0x7f2bdc1541d0 addr v1:172.21.15.18:6790/0 2024-03-29T11:11:56.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.215+0000 7f2be176d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bdc141b30 con 0x7f2bdc0b1f60 2024-03-29T11:11:56.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.215+0000 7f2be176d640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bdc141d80 con 0x7f2bdc143d10 2024-03-29T11:11:56.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.215+0000 7f2be176d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bdc142060 con 0x7f2bdc1541d0 2024-03-29T11:11:56.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.215+0000 7f2be176d640 10 monclient(hunting): _renew_subs 2024-03-29T11:11:56.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.215+0000 7f2be176d640 10 monclient(hunting): authenticate will time out at 942.487073s 2024-03-29T11:11:56.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.216+0000 7f2bda7fc640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f2bdc143d10 0x7f2bdc154c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.18:44544/0 (socket says 172.21.15.18:44544) 2024-03-29T11:11:56.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.216+0000 7f2bda7fc640 1 -- 172.21.15.18:0/2434645904 learned_addr learned my addr 172.21.15.18:0/2434645904 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:11:56.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.218+0000 7f2bc37fe640 10 client.?.objecter ms_handle_connect 0x7f2bdc143d10 2024-03-29T11:11:56.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.218+0000 7f2bc37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:56.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.219+0000 7f2bc37fe640 1 -- 172.21.15.18:0/2434645904 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 976979534 0 0) 0x7f2bdc141d80 con 0x7f2bdc143d10 2024-03-29T11:11:56.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.219+0000 7f2bc37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:11:56.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.219+0000 7f2bc37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:56.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.219+0000 7f2bc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:56.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.219+0000 7f2bc37fe640 10 monclient(hunting): my global_id is 4330 2024-03-29T11:11:56.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.219+0000 7f2bc37fe640 1 -- 172.21.15.18:0/2434645904 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bc8004eb0 con 0x7f2bdc143d10 2024-03-29T11:11:56.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.219+0000 7f2bc37fe640 10 client.?.objecter ms_handle_connect 0x7f2bdc0b1f60 2024-03-29T11:11:56.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.219+0000 7f2bc37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:56.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.219+0000 7f2bc37fe640 10 client.?.objecter ms_handle_connect 0x7f2bdc1541d0 2024-03-29T11:11:56.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.219+0000 7f2bc37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:11:56.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.219+0000 7f2bdaffd640 1 -- 172.21.15.18:0/2434645904 >> v1:172.21.15.18:6789/0 conn(0x7f2bdc0b1f60 legacy=0x7f2bdc0710e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-29T11:11:56.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.219+0000 7f2bdaffd640 1 -- 172.21.15.18:0/2434645904 >> v1:172.21.15.18:6789/0 conn(0x7f2bdc0b1f60 legacy=0x7f2bdc0710e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:11:56.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.219+0000 7f2bdaffd640 1 --1- 172.21.15.18:0/2434645904 >> v1:172.21.15.18:6789/0 conn(0x7f2bdc0b1f60 0x7f2bdc0710e0 :-1 s=OPENED pgs=281642 cs=1 l=1).handle_message read tag failed 2024-03-29T11:11:56.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.219+0000 7f2bdaffd640 1 --1- 172.21.15.18:0/2434645904 >> v1:172.21.15.18:6789/0 conn(0x7f2bdc0b1f60 0x7f2bdc0710e0 :-1 s=OPENED pgs=281642 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:11:56.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.220+0000 7f2bc37fe640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.18:6789/0 2024-03-29T11:11:56.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.220+0000 7f2bc37fe640 1 -- 172.21.15.18:0/2434645904 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1780869990 0 0) 0x7f2bdc142060 con 0x7f2bdc1541d0 2024-03-29T11:11:56.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.220+0000 7f2bc37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:11:56.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.220+0000 7f2bc37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:11:56.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.220+0000 7f2bc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:11:56.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.220+0000 7f2bc37fe640 10 monclient(hunting): my global_id is 4349 2024-03-29T11:11:56.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.220+0000 7f2bc37fe640 1 -- 172.21.15.18:0/2434645904 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2bdc141d80 con 0x7f2bdc1541d0 2024-03-29T11:11:56.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.220+0000 7f2bc37fe640 1 -- 172.21.15.18:0/2434645904 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2210425777 0 0) 0x7f2bc8004eb0 con 0x7f2bdc143d10 2024-03-29T11:11:56.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.220+0000 7f2bc37fe640 1 -- 172.21.15.18:0/2434645904 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2bc8005490 con 0x7f2bdc143d10 2024-03-29T11:11:56.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.220+0000 7f2bc37fe640 1 -- 172.21.15.18:0/2434645904 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1903945869 0 0) 0x7f2bdc141d80 con 0x7f2bdc1541d0 2024-03-29T11:11:56.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.220+0000 7f2bc37fe640 1 -- 172.21.15.18:0/2434645904 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2bc8004eb0 con 0x7f2bdc1541d0 2024-03-29T11:11:56.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.220+0000 7f2bc37fe640 1 -- 172.21.15.18:0/2434645904 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f2bd4001ef0 con 0x7f2bdc1541d0 2024-03-29T11:11:56.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.220+0000 7f2bc37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:56.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.220+0000 7f2bc37fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:11:56.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.220+0000 7f2bc37fe640 10 monclient(hunting): dump: 2024-03-29T11:11:56.208 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:11:56.208 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:56.208 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:56.208 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:56.208 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:56.208 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:56.209 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:11:56.209 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:11:56.209 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:11:56.209 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:56.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.220+0000 7f2bc37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:11:56.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.220+0000 7f2bc37fe640 1 -- 172.21.15.18:0/2434645904 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f2bcc004080 con 0x7f2bdc143d10 2024-03-29T11:11:56.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.220+0000 7f2bc37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:56.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.220+0000 7f2bc37fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:11:56.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.220+0000 7f2bc37fe640 10 monclient(hunting): dump: 2024-03-29T11:11:56.210 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:11:56.210 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:56.210 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:56.210 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:56.211 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:56.211 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:56.211 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:11:56.211 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:11:56.211 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:11:56.211 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:56.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.221+0000 7f2bc37fe640 1 -- 172.21.15.18:0/2434645904 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 487551082 0 0) 0x7f2bc8004eb0 con 0x7f2bdc1541d0 2024-03-29T11:11:56.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.221+0000 7f2bc37fe640 1 -- 172.21.15.18:0/2434645904 >> v1:172.21.15.70:6789/0 conn(0x7f2bdc143d10 legacy=0x7f2bdc154c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:56.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.221+0000 7f2bc37fe640 1 -- 172.21.15.18:0/2434645904 >> v1:172.21.15.18:6789/0 conn(0x7f2bdc0b1f60 legacy=0x7f2bdc0710e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:11:56.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.221+0000 7f2bc37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:11:56.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.221+0000 7f2bc37fe640 1 monclient: found mon.c 2024-03-29T11:11:56.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.221+0000 7f2bc37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:11:56.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.221+0000 7f2bc37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:11:56.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.221+0000 7f2bc37fe640 1 -- 172.21.15.18:0/2434645904 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bdc142760 con 0x7f2bdc1541d0 2024-03-29T11:11:56.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.221+0000 7f2bc37fe640 10 monclient: _finish_auth 0 2024-03-29T11:11:56.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.221+0000 7f2bc37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:11:56.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.221+0000 7f2bc37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:11:56.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.221+0000 7f2be176d640 5 monclient: authenticate success, global_id 4349 2024-03-29T11:11:56.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.221+0000 7f2be176d640 10 monclient: _renew_subs 2024-03-29T11:11:56.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.221+0000 7f2be176d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:11:56.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.221+0000 7f2be176d640 1 -- 172.21.15.18:0/2434645904 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bdc141090 con 0x7f2bdc1541d0 2024-03-29T11:11:56.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.221+0000 7f2be176d640 10 client.4349.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:11:56.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.221+0000 7f2be176d640 10 monclient: _renew_subs 2024-03-29T11:11:56.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.221+0000 7f2be176d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:11:56.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.221+0000 7f2be176d640 1 -- 172.21.15.18:0/2434645904 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bdc1409e0 con 0x7f2bdc1541d0 2024-03-29T11:11:56.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.221+0000 7f2be176d640 1 librados: init done 2024-03-29T11:11:56.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.221+0000 7f2bc37fe640 1 -- 172.21.15.18:0/2434645904 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2bd4002050 con 0x7f2bdc1541d0 2024-03-29T11:11:56.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.221+0000 7f2bc37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:11:56.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.221+0000 7f2bc37fe640 1 -- 172.21.15.18:0/2434645904 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f2bd4004fb0 con 0x7f2bdc1541d0 2024-03-29T11:11:56.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.221+0000 7f2bc37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:11:56.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.221+0000 7f2bc37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:11:56.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.221+0000 7f2bd97fa640 4 set_mon_vals no callback set 2024-03-29T11:11:56.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.221+0000 7f2bc37fe640 10 monclient: dump: 2024-03-29T11:11:56.216 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:11:56.216 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:11:56.216 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:56.216 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:11:56.217 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:11:56.217 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:11:56.217 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:11:56.217 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:11:56.217 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:11:56.217 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:11:56.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.222+0000 7f2be176d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:56.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.223+0000 7f2be176d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:56.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.223+0000 7f2be176d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:11:56.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.223+0000 7f2be176d640 1 -- 172.21.15.18:0/2434645904 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bdc075f10 con 0x7f2bdc1541d0 2024-03-29T11:11:56.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.223+0000 7f2bc37fe640 1 -- 172.21.15.18:0/2434645904 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f2bd401af80 con 0x7f2bdc1541d0 2024-03-29T11:11:56.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.224+0000 7f2bc37fe640 1 -- 172.21.15.18:0/2434645904 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f2bd4084b20 con 0x7f2bdc1541d0 2024-03-29T11:11:56.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.224+0000 7f2bc37fe640 10 client.4349.objecter ms_dispatch 0x7f2bdc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:11:56.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.224+0000 7f2bc37fe640 3 client.4349.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:11:56.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.224+0000 7f2bc37fe640 3 client.4349.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:11:56.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.224+0000 7f2bc37fe640 20 client.4349.objecter dump_active .. 0 homeless 2024-03-29T11:11:56.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.224+0000 7f2bdaffd640 10 monclient: get_auth_request con 0x7f2bc806e1c0 auth_method 0 2024-03-29T11:11:56.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.225+0000 7f2bc37fe640 10 client.4349.objecter ms_handle_connect 0x7f2bc806e1c0 2024-03-29T11:11:56.221 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.231+0000 7f2bc37fe640 1 -- 172.21.15.18:0/2434645904 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2bd4050400 con 0x7f2bdc1541d0 2024-03-29T11:11:56.221 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.231+0000 7f2bc37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:11:56.221 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.231+0000 7f2bc37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:11:56.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.556+0000 7f2be176d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:56.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.557+0000 7f2be176d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:56.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.557+0000 7f2be176d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:11:56.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.557+0000 7f2be176d640 1 -- 172.21.15.18:0/2434645904 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2bdc141220 con 0x7f2bdc1541d0 2024-03-29T11:11:56.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.557+0000 7f2bc37fe640 1 -- 172.21.15.18:0/2434645904 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f2bd4004e10 con 0x7f2bdc1541d0 2024-03-29T11:11:56.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.557+0000 7f2bc37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:11:56.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.557+0000 7f2bc37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:11:56.538 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:11:56.538 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:11:56.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.561+0000 7f2be176d640 10 librados: watch_flush enter 2024-03-29T11:11:56.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.561+0000 7f2be176d640 10 librados: watch_flush exit 2024-03-29T11:11:56.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.562+0000 7f2be176d640 20 client.4349.objecter shutdown clearing up homeless session... 2024-03-29T11:11:56.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.562+0000 7f2be176d640 10 client.4349.objecter successfully canceled tick 2024-03-29T11:11:56.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.562+0000 7f2be176d640 1 -- 172.21.15.18:0/2434645904 >> v1:172.21.15.70:6816/3101871562 conn(0x7f2bc806e1c0 legacy=0x7f2bc8070680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:56.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.562+0000 7f2be176d640 10 monclient: shutdown 2024-03-29T11:11:56.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.562+0000 7f2be176d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:11:56.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.562+0000 7f2be176d640 1 -- 172.21.15.18:0/2434645904 >> v1:172.21.15.18:6790/0 conn(0x7f2bdc1541d0 legacy=0x7f2bdc1eed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:11:56.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.562+0000 7f2be176d640 1 -- 172.21.15.18:0/2434645904 shutdown_connections 2024-03-29T11:11:56.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.562+0000 7f2be176d640 1 -- 172.21.15.18:0/2434645904 >> 172.21.15.18:0/2434645904 conn(0x7f2bdc066a00 msgr2=0x7f2bdc1468c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:11:56.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.562+0000 7f2be176d640 1 -- 172.21.15.18:0/2434645904 shutdown_connections 2024-03-29T11:11:56.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.562+0000 7f2be176d640 1 -- 172.21.15.18:0/2434645904 wait complete. 2024-03-29T11:11:56.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:11:56.562+0000 7f2be176d640 1 librados: shutdown 2024-03-29T11:11:56.556 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:12:03.362 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:03.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.532+0000 7f566bf96640 10 monclient: get_monmap_and_config 2024-03-29T11:12:03.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.532+0000 7f566bf96640 10 monclient: build_initial_monmap 2024-03-29T11:12:03.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.533+0000 7f566bf96640 10 monclient: monmap: 2024-03-29T11:12:03.514 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:12:03.514 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:03.514 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:12:03.533892+0000 2024-03-29T11:12:03.514 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:12:03.533892+0000 2024-03-29T11:12:03.515 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:03.515 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:03.515 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:12:03.515 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:12:03.515 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:12:03.515 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:03.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.537+0000 7f566bf96640 1 Processor -- start 2024-03-29T11:12:03.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.537+0000 7f566bf96640 1 -- start start 2024-03-29T11:12:03.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.537+0000 7f566bf96640 10 monclient: init 2024-03-29T11:12:03.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.538+0000 7f566bf96640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:03.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.538+0000 7f566bf96640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:12:03.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.538+0000 7f566bf96640 10 monclient(hunting): picked mon.noname-c con 0x7f56640a9930 addr v1:172.21.15.70:6789/0 2024-03-29T11:12:03.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.538+0000 7f566bf96640 10 monclient(hunting): picked mon.noname-b con 0x7f5664149dc0 addr v1:172.21.15.18:6790/0 2024-03-29T11:12:03.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.538+0000 7f566bf96640 10 monclient(hunting): picked mon.noname-a con 0x7f566414ff40 addr v1:172.21.15.18:6789/0 2024-03-29T11:12:03.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.538+0000 7f566bf96640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56640aae60 con 0x7f566414ff40 2024-03-29T11:12:03.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.538+0000 7f566bf96640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56640b4310 con 0x7f56640a9930 2024-03-29T11:12:03.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.538+0000 7f566bf96640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56640afdd0 con 0x7f5664149dc0 2024-03-29T11:12:03.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.538+0000 7f566bf96640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:03.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.538+0000 7f566bf96640 10 monclient(hunting): authenticate will time out at 949.810260s 2024-03-29T11:12:03.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.540+0000 7f5669531640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f5664149dc0 0x7f566414a240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:35575/0 (socket says 172.21.15.18:35575) 2024-03-29T11:12:03.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.540+0000 7f5669531640 1 -- 172.21.15.18:0/654257971 learned_addr learned my addr 172.21.15.18:0/654257971 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:12:03.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.545+0000 7f5668d30640 1 -- 172.21.15.18:0/654257971 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 510950223 0 0) 0x7f56640afdd0 con 0x7f5664149dc0 2024-03-29T11:12:03.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.545+0000 7f5668d30640 10 monclient(hunting): _negotiate 2024-03-29T11:12:03.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.545+0000 7f5668d30640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:03.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.545+0000 7f5668d30640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:03.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.545+0000 7f5668d30640 10 monclient(hunting): my global_id is 4352 2024-03-29T11:12:03.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.545+0000 7f5668d30640 1 -- 172.21.15.18:0/654257971 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5650003520 con 0x7f5664149dc0 2024-03-29T11:12:03.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.545+0000 7f5668d30640 1 -- 172.21.15.18:0/654257971 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3010452514 0 0) 0x7f5650003520 con 0x7f5664149dc0 2024-03-29T11:12:03.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.545+0000 7f5668d30640 1 -- 172.21.15.18:0/654257971 >> v1:172.21.15.70:6789/0 conn(0x7f56640a9930 legacy=0x7f5664143a30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:12:03.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.545+0000 7f5668d30640 1 -- 172.21.15.18:0/654257971 >> v1:172.21.15.18:6789/0 conn(0x7f566414ff40 legacy=0x7f5664152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:03.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.545+0000 7f5668d30640 10 monclient: _finish_hunting 0 2024-03-29T11:12:03.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.545+0000 7f5668d30640 1 monclient: found mon.noname-b 2024-03-29T11:12:03.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.545+0000 7f5668d30640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:03.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.545+0000 7f5668d30640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:12:03.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.545+0000 7f5668d30640 1 -- 172.21.15.18:0/654257971 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f566414a950 con 0x7f5664149dc0 2024-03-29T11:12:03.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.545+0000 7f5668d30640 10 monclient: _finish_auth 0 2024-03-29T11:12:03.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.545+0000 7f5668d30640 10 monclient: _check_auth_tickets 2024-03-29T11:12:03.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.545+0000 7f5668d30640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:03.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.546+0000 7f566bf96640 5 monclient: authenticate success, global_id 4352 2024-03-29T11:12:03.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.546+0000 7f566bf96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:03.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.546+0000 7f5668d30640 1 -- 172.21.15.18:0/654257971 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f565c001690 con 0x7f5664149dc0 2024-03-29T11:12:03.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.546+0000 7f5668d30640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:03.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.546+0000 7f5668d30640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:03.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.546+0000 7f5668d30640 10 monclient: dump: 2024-03-29T11:12:03.528 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:12:03.528 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:03.528 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:03.528 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:03.528 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:03.529 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:03.529 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:12:03.529 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:12:03.529 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:12:03.529 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:03.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.546+0000 7f566bf96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:03.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.546+0000 7f5668d30640 1 -- 172.21.15.18:0/654257971 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f565c001920 con 0x7f5664149dc0 2024-03-29T11:12:03.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.546+0000 7f5668d30640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:03.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.546+0000 7f566bf96640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:03.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.546+0000 7f566bf96640 4 set_mon_vals no callback set 2024-03-29T11:12:03.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.547+0000 7f566bf96640 10 monclient: shutdown 2024-03-29T11:12:03.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.547+0000 7f566bf96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:03.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.547+0000 7f566bf96640 1 -- 172.21.15.18:0/654257971 >> v1:172.21.15.18:6790/0 conn(0x7f5664149dc0 legacy=0x7f566414a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.547+0000 7f566bf96640 1 -- 172.21.15.18:0/654257971 shutdown_connections 2024-03-29T11:12:03.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.547+0000 7f566bf96640 1 -- 172.21.15.18:0/654257971 >> 172.21.15.18:0/654257971 conn(0x7f56640b35e0 msgr2=0x7f56640b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:03.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.550+0000 7f566bf96640 1 -- 172.21.15.18:0/654257971 shutdown_connections 2024-03-29T11:12:03.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.550+0000 7f566bf96640 1 -- 172.21.15.18:0/654257971 wait complete. 2024-03-29T11:12:03.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.551+0000 7f566bf96640 10 monclient: build_initial_monmap 2024-03-29T11:12:03.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.551+0000 7f566bf96640 10 monclient: monmap: 2024-03-29T11:12:03.530 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:12:03.530 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:03.530 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:12:03.531 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:12:03.531 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:03.531 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:03.531 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:12:03.531 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:12:03.531 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:12:03.531 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:03.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.551+0000 7f566bf96640 1 librados: starting msgr at 2024-03-29T11:12:03.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.551+0000 7f566bf96640 1 librados: starting objecter 2024-03-29T11:12:03.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.552+0000 7f566bf96640 1 Processor -- start 2024-03-29T11:12:03.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.552+0000 7f566bf96640 1 -- start start 2024-03-29T11:12:03.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.552+0000 7f566bf96640 1 librados: setting wanted keys 2024-03-29T11:12:03.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.552+0000 7f566bf96640 1 librados: calling monclient init 2024-03-29T11:12:03.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.552+0000 7f566bf96640 10 monclient: init 2024-03-29T11:12:03.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.552+0000 7f566bf96640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:03.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.552+0000 7f566bf96640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:12:03.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.552+0000 7f566bf96640 10 monclient(hunting): picked mon.noname-a con 0x7f56640a9930 addr v1:172.21.15.18:6789/0 2024-03-29T11:12:03.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.553+0000 7f566bf96640 10 monclient(hunting): picked mon.noname-b con 0x7f5664149dc0 addr v1:172.21.15.70:6789/0 2024-03-29T11:12:03.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.553+0000 7f566bf96640 10 monclient(hunting): picked mon.noname-c con 0x7f566414ff40 addr v1:172.21.15.18:6790/0 2024-03-29T11:12:03.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.553+0000 7f566bf96640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f566414aae0 con 0x7f56640a9930 2024-03-29T11:12:03.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.553+0000 7f566bf96640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5664071430 con 0x7f5664149dc0 2024-03-29T11:12:03.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.553+0000 7f566bf96640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5664132140 con 0x7f566414ff40 2024-03-29T11:12:03.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.553+0000 7f566bf96640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:03.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.553+0000 7f566bf96640 10 monclient(hunting): authenticate will time out at 949.824975s 2024-03-29T11:12:03.535 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.556+0000 7f5669531640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f5664149dc0 0x7f566414e600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.18:41017/0 (socket says 172.21.15.18:41017) 2024-03-29T11:12:03.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.556+0000 7f5669531640 1 -- 172.21.15.18:0/1523226773 learned_addr learned my addr 172.21.15.18:0/1523226773 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:12:03.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.559+0000 7f565a7fc640 10 client.?.objecter ms_handle_connect 0x7f5664149dc0 2024-03-29T11:12:03.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.559+0000 7f565a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:03.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.559+0000 7f565a7fc640 1 -- 172.21.15.18:0/1523226773 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 982972454 0 0) 0x7f5664071430 con 0x7f5664149dc0 2024-03-29T11:12:03.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.559+0000 7f565a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:12:03.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.559+0000 7f565a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:03.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.559+0000 7f565a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:03.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.559+0000 7f565a7fc640 10 monclient(hunting): my global_id is 4333 2024-03-29T11:12:03.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.559+0000 7f565a7fc640 1 -- 172.21.15.18:0/1523226773 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5648004eb0 con 0x7f5664149dc0 2024-03-29T11:12:03.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.560+0000 7f565a7fc640 10 client.?.objecter ms_handle_connect 0x7f566414ff40 2024-03-29T11:12:03.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.560+0000 7f565a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:03.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.560+0000 7f565a7fc640 10 client.?.objecter ms_handle_connect 0x7f56640a9930 2024-03-29T11:12:03.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.560+0000 7f565a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:03.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.560+0000 7f565a7fc640 1 -- 172.21.15.18:0/1523226773 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2205651110 0 0) 0x7f5664132140 con 0x7f566414ff40 2024-03-29T11:12:03.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.560+0000 7f565a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:12:03.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.560+0000 7f565a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:03.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.560+0000 7f565a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:03.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.560+0000 7f565a7fc640 10 monclient(hunting): my global_id is 4355 2024-03-29T11:12:03.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.560+0000 7f565a7fc640 1 -- 172.21.15.18:0/1523226773 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5664071430 con 0x7f566414ff40 2024-03-29T11:12:03.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.560+0000 7f565a7fc640 1 -- 172.21.15.18:0/1523226773 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2274340274 0 0) 0x7f5648004eb0 con 0x7f5664149dc0 2024-03-29T11:12:03.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.560+0000 7f565a7fc640 1 -- 172.21.15.18:0/1523226773 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5664132140 con 0x7f5664149dc0 2024-03-29T11:12:03.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.560+0000 7f565a7fc640 1 -- 172.21.15.18:0/1523226773 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f565c015b30 con 0x7f5664149dc0 2024-03-29T11:12:03.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.560+0000 7f565a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:03.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.560+0000 7f565a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:03.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.560+0000 7f565a7fc640 10 monclient(hunting): dump: 2024-03-29T11:12:03.542 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:12:03.542 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:03.542 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:03.542 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:03.542 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:03.543 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:03.543 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:12:03.543 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:12:03.543 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:12:03.543 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:03.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.560+0000 7f565a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:03.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.561+0000 7f565a7fc640 1 -- 172.21.15.18:0/1523226773 <== mon.1 v1:172.21.15.70:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1856534362 0 0) 0x7f5664132140 con 0x7f5664149dc0 2024-03-29T11:12:03.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.561+0000 7f565a7fc640 1 -- 172.21.15.18:0/1523226773 >> v1:172.21.15.18:6790/0 conn(0x7f566414ff40 legacy=0x7f566406e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.561+0000 7f565a7fc640 1 -- 172.21.15.18:0/1523226773 >> v1:172.21.15.18:6789/0 conn(0x7f56640a9930 legacy=0x7f5664131a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.561+0000 7f565a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:12:03.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.561+0000 7f565a7fc640 1 monclient: found mon.b 2024-03-29T11:12:03.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.561+0000 7f565a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:03.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.561+0000 7f565a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:12:03.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.561+0000 7f565a7fc640 1 -- 172.21.15.18:0/1523226773 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f566414b920 con 0x7f5664149dc0 2024-03-29T11:12:03.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.561+0000 7f565a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:12:03.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.561+0000 7f565a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:12:03.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.561+0000 7f565a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:03.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.562+0000 7f566bf96640 5 monclient: authenticate success, global_id 4333 2024-03-29T11:12:03.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.562+0000 7f566bf96640 10 monclient: _renew_subs 2024-03-29T11:12:03.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.562+0000 7f566bf96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:12:03.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.562+0000 7f566bf96640 1 -- 172.21.15.18:0/1523226773 --> v1:172.21.15.70:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f566414bc00 con 0x7f5664149dc0 2024-03-29T11:12:03.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.562+0000 7f566bf96640 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:03.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.562+0000 7f566bf96640 10 monclient: _renew_subs 2024-03-29T11:12:03.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.562+0000 7f566bf96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:12:03.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.562+0000 7f566bf96640 1 -- 172.21.15.18:0/1523226773 --> v1:172.21.15.70:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f566414ed10 con 0x7f5664149dc0 2024-03-29T11:12:03.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.562+0000 7f566bf96640 1 librados: init done 2024-03-29T11:12:03.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.563+0000 7f565a7fc640 1 -- 172.21.15.18:0/1523226773 <== mon.1 v1:172.21.15.70:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f565c015f40 con 0x7f5664149dc0 2024-03-29T11:12:03.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.563+0000 7f565a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:03.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.563+0000 7f565a7fc640 1 -- 172.21.15.18:0/1523226773 <== mon.1 v1:172.21.15.70:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f565c016390 con 0x7f5664149dc0 2024-03-29T11:12:03.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.563+0000 7f565a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:03.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.563+0000 7f565a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:03.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.563+0000 7f565a7fc640 10 monclient: dump: 2024-03-29T11:12:03.545 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:12:03.545 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:03.545 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:03.545 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:03.545 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:03.545 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:03.545 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:12:03.545 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:12:03.545 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:12:03.545 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:03.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.563+0000 7f565bfff640 4 set_mon_vals no callback set 2024-03-29T11:12:03.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.564+0000 7f566bf96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:03.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.564+0000 7f565a7fc640 1 -- 172.21.15.18:0/1523226773 <== mon.1 v1:172.21.15.70:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f565c001b50 con 0x7f5664149dc0 2024-03-29T11:12:03.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.565+0000 7f566bf96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:03.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.565+0000 7f566bf96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:12:03.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.565+0000 7f566bf96640 1 -- 172.21.15.18:0/1523226773 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f564c005080 con 0x7f5664149dc0 2024-03-29T11:12:03.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.566+0000 7f565a7fc640 1 -- 172.21.15.18:0/1523226773 <== mon.1 v1:172.21.15.70:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f565c095da0 con 0x7f5664149dc0 2024-03-29T11:12:03.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.566+0000 7f565a7fc640 10 client.4333.objecter ms_dispatch 0x7f5664000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:12:03.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.566+0000 7f565a7fc640 3 client.4333.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:12:03.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.566+0000 7f565a7fc640 3 client.4333.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:12:03.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.566+0000 7f565a7fc640 20 client.4333.objecter dump_active .. 0 homeless 2024-03-29T11:12:03.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.567+0000 7f5669d32640 10 monclient: get_auth_request con 0x7f564806de40 auth_method 0 2024-03-29T11:12:03.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.568+0000 7f565a7fc640 10 client.4333.objecter ms_handle_connect 0x7f564806de40 2024-03-29T11:12:03.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.574+0000 7f565a7fc640 1 -- 172.21.15.18:0/1523226773 <== mon.1 v1:172.21.15.70:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f565c061680 con 0x7f5664149dc0 2024-03-29T11:12:03.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.574+0000 7f565a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:03.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.574+0000 7f565a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:03.881 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.901+0000 7f566bf96640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:03.881 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.902+0000 7f566bf96640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:03.881 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.902+0000 7f566bf96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:12:03.881 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.902+0000 7f566bf96640 1 -- 172.21.15.18:0/1523226773 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f564c005330 con 0x7f5664149dc0 2024-03-29T11:12:03.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.907+0000 7f565a7fc640 1 -- 172.21.15.18:0/1523226773 <== mon.1 v1:172.21.15.70:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f565c0635b0 con 0x7f5664149dc0 2024-03-29T11:12:03.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.907+0000 7f565a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:03.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.907+0000 7f565a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:03.886 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:12:03.886 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:03.890 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.910+0000 7f566bf96640 10 librados: watch_flush enter 2024-03-29T11:12:03.890 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.912+0000 7f566bf96640 10 librados: watch_flush exit 2024-03-29T11:12:03.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.912+0000 7f566bf96640 20 client.4333.objecter shutdown clearing up homeless session... 2024-03-29T11:12:03.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.912+0000 7f566bf96640 10 client.4333.objecter successfully canceled tick 2024-03-29T11:12:03.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.912+0000 7f566bf96640 1 -- 172.21.15.18:0/1523226773 >> v1:172.21.15.70:6816/3101871562 conn(0x7f564806de40 legacy=0x7f5648070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.912+0000 7f566bf96640 10 monclient: shutdown 2024-03-29T11:12:03.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.912+0000 7f566bf96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:03.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.912+0000 7f566bf96640 1 -- 172.21.15.18:0/1523226773 >> v1:172.21.15.70:6789/0 conn(0x7f5664149dc0 legacy=0x7f566414e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:03.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.913+0000 7f566bf96640 1 -- 172.21.15.18:0/1523226773 shutdown_connections 2024-03-29T11:12:03.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.913+0000 7f566bf96640 1 -- 172.21.15.18:0/1523226773 >> 172.21.15.18:0/1523226773 conn(0x7f56640b35e0 msgr2=0x7f56640b2770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:03.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.914+0000 7f566bf96640 1 -- 172.21.15.18:0/1523226773 shutdown_connections 2024-03-29T11:12:03.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.914+0000 7f566bf96640 1 -- 172.21.15.18:0/1523226773 wait complete. 2024-03-29T11:12:03.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:03.914+0000 7f566bf96640 1 librados: shutdown 2024-03-29T11:12:03.910 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:12:10.717 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:10.873 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.894+0000 7f900bee2640 10 monclient: get_monmap_and_config 2024-03-29T11:12:10.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.894+0000 7f900bee2640 10 monclient: build_initial_monmap 2024-03-29T11:12:10.874 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.894+0000 7f900bee2640 10 monclient: monmap: 2024-03-29T11:12:10.874 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:12:10.874 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:10.874 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:12:10.895559+0000 2024-03-29T11:12:10.874 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:12:10.895559+0000 2024-03-29T11:12:10.874 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:10.874 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:10.874 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:12:10.874 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:12:10.874 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:12:10.875 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:10.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.895+0000 7f900bee2640 1 Processor -- start 2024-03-29T11:12:10.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.895+0000 7f900bee2640 1 -- start start 2024-03-29T11:12:10.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.895+0000 7f900bee2640 10 monclient: init 2024-03-29T11:12:10.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.896+0000 7f900bee2640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:10.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.896+0000 7f900bee2640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:12:10.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.896+0000 7f900bee2640 10 monclient(hunting): picked mon.noname-a con 0x7f90040ad6c0 addr v1:172.21.15.18:6789/0 2024-03-29T11:12:10.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.896+0000 7f900bee2640 10 monclient(hunting): picked mon.noname-c con 0x7f9004152980 addr v1:172.21.15.70:6789/0 2024-03-29T11:12:10.875 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.896+0000 7f900bee2640 10 monclient(hunting): picked mon.noname-b con 0x7f90041588d0 addr v1:172.21.15.18:6790/0 2024-03-29T11:12:10.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.896+0000 7f900bee2640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90040a94e0 con 0x7f90040ad6c0 2024-03-29T11:12:10.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.896+0000 7f900bee2640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90040af3a0 con 0x7f9004152980 2024-03-29T11:12:10.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.896+0000 7f900bee2640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90040afb60 con 0x7f90041588d0 2024-03-29T11:12:10.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.896+0000 7f900bee2640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:10.876 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.896+0000 7f900bee2640 10 monclient(hunting): authenticate will time out at 957.168622s 2024-03-29T11:12:10.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.900+0000 7f900a47f640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f90041588d0 0x7f900415acc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:55853/0 (socket says 172.21.15.18:55853) 2024-03-29T11:12:10.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.900+0000 7f900a47f640 1 -- 172.21.15.18:0/4147520409 learned_addr learned my addr 172.21.15.18:0/4147520409 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:12:10.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.903+0000 7f9008c7c640 1 -- 172.21.15.18:0/4147520409 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1469741480 0 0) 0x7f90040afb60 con 0x7f90041588d0 2024-03-29T11:12:10.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.903+0000 7f9008c7c640 10 monclient(hunting): _negotiate 2024-03-29T11:12:10.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.903+0000 7f9008c7c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:10.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.903+0000 7f9008c7c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:10.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.903+0000 7f9008c7c640 10 monclient(hunting): my global_id is 4358 2024-03-29T11:12:10.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.903+0000 7f9008c7c640 1 -- 172.21.15.18:0/4147520409 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ff0003520 con 0x7f90041588d0 2024-03-29T11:12:10.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.904+0000 7f9008c7c640 1 -- 172.21.15.18:0/4147520409 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 43042421 0 0) 0x7f8ff0003520 con 0x7f90041588d0 2024-03-29T11:12:10.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.904+0000 7f9008c7c640 1 -- 172.21.15.18:0/4147520409 >> v1:172.21.15.70:6789/0 conn(0x7f9004152980 legacy=0x7f9004152df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:10.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.904+0000 7f9008c7c640 1 -- 172.21.15.18:0/4147520409 >> v1:172.21.15.18:6789/0 conn(0x7f90040ad6c0 legacy=0x7f900414fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:10.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.904+0000 7f9008c7c640 10 monclient: _finish_hunting 0 2024-03-29T11:12:10.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.904+0000 7f9008c7c640 1 monclient: found mon.noname-b 2024-03-29T11:12:10.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.904+0000 7f9008c7c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:10.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.904+0000 7f9008c7c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:12:10.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.904+0000 7f9008c7c640 1 -- 172.21.15.18:0/4147520409 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90040aab80 con 0x7f90041588d0 2024-03-29T11:12:10.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.904+0000 7f9008c7c640 10 monclient: _finish_auth 0 2024-03-29T11:12:10.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.904+0000 7f9008c7c640 10 monclient: _check_auth_tickets 2024-03-29T11:12:10.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.904+0000 7f9008c7c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:10.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.904+0000 7f9008c7c640 1 -- 172.21.15.18:0/4147520409 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f8ffc001a10 con 0x7f90041588d0 2024-03-29T11:12:10.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.904+0000 7f9008c7c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:10.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.904+0000 7f9008c7c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:10.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.904+0000 7f9008c7c640 10 monclient: dump: 2024-03-29T11:12:10.886 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:12:10.886 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:10.886 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:10.887 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:10.887 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:10.887 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:10.887 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:12:10.887 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:12:10.887 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:12:10.887 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:10.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.904+0000 7f900bee2640 5 monclient: authenticate success, global_id 4358 2024-03-29T11:12:10.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.904+0000 7f900bee2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:10.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.905+0000 7f9008c7c640 1 -- 172.21.15.18:0/4147520409 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ffc001b70 con 0x7f90041588d0 2024-03-29T11:12:10.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.905+0000 7f9008c7c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:10.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.905+0000 7f900bee2640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:10.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.905+0000 7f900bee2640 4 set_mon_vals no callback set 2024-03-29T11:12:10.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.905+0000 7f9008c7c640 1 -- 172.21.15.18:0/4147520409 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f8ffc001fc0 con 0x7f90041588d0 2024-03-29T11:12:10.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.905+0000 7f9008c7c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:10.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.905+0000 7f9008c7c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:10.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.905+0000 7f9008c7c640 10 monclient: dump: 2024-03-29T11:12:10.888 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:12:10.888 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:10.888 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:10.888 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:10.888 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:10.888 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:10.888 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:12:10.888 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:12:10.888 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:12:10.888 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:10.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.906+0000 7f900bee2640 10 monclient: shutdown 2024-03-29T11:12:10.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.906+0000 7f900bee2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:10.888 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.906+0000 7f900bee2640 1 -- 172.21.15.18:0/4147520409 >> v1:172.21.15.18:6790/0 conn(0x7f90041588d0 legacy=0x7f900415acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:10.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.906+0000 7f900bee2640 1 -- 172.21.15.18:0/4147520409 shutdown_connections 2024-03-29T11:12:10.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.906+0000 7f900bee2640 1 -- 172.21.15.18:0/4147520409 >> 172.21.15.18:0/4147520409 conn(0x7f90040b35e0 msgr2=0x7f90040b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:10.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.906+0000 7f900bee2640 1 -- 172.21.15.18:0/4147520409 shutdown_connections 2024-03-29T11:12:10.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.907+0000 7f900bee2640 1 -- 172.21.15.18:0/4147520409 wait complete. 2024-03-29T11:12:10.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.907+0000 7f900bee2640 10 monclient: build_initial_monmap 2024-03-29T11:12:10.890 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.907+0000 7f900bee2640 10 monclient: monmap: 2024-03-29T11:12:10.890 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:12:10.890 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:10.890 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:12:10.890 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:12:10.890 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:10.890 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:10.890 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:12:10.890 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:12:10.890 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:12:10.890 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:10.890 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.908+0000 7f900bee2640 1 librados: starting msgr at 2024-03-29T11:12:10.890 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.908+0000 7f900bee2640 1 librados: starting objecter 2024-03-29T11:12:10.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.908+0000 7f900bee2640 1 Processor -- start 2024-03-29T11:12:10.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.908+0000 7f900bee2640 1 -- start start 2024-03-29T11:12:10.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.908+0000 7f900bee2640 1 librados: setting wanted keys 2024-03-29T11:12:10.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.908+0000 7f900bee2640 1 librados: calling monclient init 2024-03-29T11:12:10.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.908+0000 7f900bee2640 10 monclient: init 2024-03-29T11:12:10.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.909+0000 7f900bee2640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:10.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.909+0000 7f900bee2640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:12:10.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.909+0000 7f900bee2640 10 monclient(hunting): picked mon.noname-a con 0x7f90040ad6c0 addr v1:172.21.15.18:6789/0 2024-03-29T11:12:10.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.909+0000 7f900bee2640 10 monclient(hunting): picked mon.noname-b con 0x7f9004152980 addr v1:172.21.15.70:6789/0 2024-03-29T11:12:10.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.909+0000 7f900bee2640 10 monclient(hunting): picked mon.noname-c con 0x7f90041588d0 addr v1:172.21.15.18:6790/0 2024-03-29T11:12:10.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.909+0000 7f900bee2640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9004070410 con 0x7f90040ad6c0 2024-03-29T11:12:10.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.909+0000 7f900bee2640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90040711c0 con 0x7f9004152980 2024-03-29T11:12:10.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.909+0000 7f900bee2640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9004071830 con 0x7f90041588d0 2024-03-29T11:12:10.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.909+0000 7f900bee2640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:10.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.909+0000 7f900bee2640 10 monclient(hunting): authenticate will time out at 957.181206s 2024-03-29T11:12:10.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.914+0000 7f9009c7e640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f90040ad6c0 0x7f900406f020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:38893/0 (socket says 172.21.15.18:38893) 2024-03-29T11:12:10.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.914+0000 7f9009c7e640 1 -- 172.21.15.18:0/1853784838 learned_addr learned my addr 172.21.15.18:0/1853784838 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:12:10.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.915+0000 7f8ffa7fc640 10 client.?.objecter ms_handle_connect 0x7f90040ad6c0 2024-03-29T11:12:10.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.915+0000 7f8ffa7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:10.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.915+0000 7f8ffa7fc640 1 -- 172.21.15.18:0/1853784838 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1101787328 0 0) 0x7f9000001270 con 0x7f90040ad6c0 2024-03-29T11:12:10.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.915+0000 7f8ffa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:12:10.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.915+0000 7f8ffa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:10.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.915+0000 7f8ffa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:10.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.915+0000 7f8ffa7fc640 10 monclient(hunting): my global_id is 4329 2024-03-29T11:12:10.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.915+0000 7f8ffa7fc640 1 -- 172.21.15.18:0/1853784838 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fe8004eb0 con 0x7f90040ad6c0 2024-03-29T11:12:10.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.916+0000 7f8ffa7fc640 1 -- 172.21.15.18:0/1853784838 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2961636478 0 0) 0x7f8fe8004eb0 con 0x7f90040ad6c0 2024-03-29T11:12:10.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.916+0000 7f8ffa7fc640 1 -- 172.21.15.18:0/1853784838 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8fe8005300 con 0x7f90040ad6c0 2024-03-29T11:12:10.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.916+0000 7f8ffa7fc640 1 -- 172.21.15.18:0/1853784838 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f90000019e0 con 0x7f90040ad6c0 2024-03-29T11:12:10.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.916+0000 7f8ffa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:10.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.916+0000 7f8ffa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:12:10.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.916+0000 7f8ffa7fc640 10 monclient(hunting): dump: 2024-03-29T11:12:10.895 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:12:10.896 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:10.896 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:10.896 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:10.896 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:10.896 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:10.896 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:12:10.897 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:12:10.897 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:12:10.897 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:10.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.916+0000 7f8ffa7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:10.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.916+0000 7f8ffa7fc640 1 -- 172.21.15.18:0/1853784838 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2764551119 0 0) 0x7f8fe8005300 con 0x7f90040ad6c0 2024-03-29T11:12:10.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.916+0000 7f8ffa7fc640 1 -- 172.21.15.18:0/1853784838 >> v1:172.21.15.18:6790/0 conn(0x7f90041588d0 legacy=0x7f900414c710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:12:10.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.917+0000 7f8ffa7fc640 1 -- 172.21.15.18:0/1853784838 >> v1:172.21.15.70:6789/0 conn(0x7f9004152980 legacy=0x7f9004071d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:12:10.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.921+0000 7f8ffa7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:12:10.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.921+0000 7f8ffa7fc640 1 monclient: found mon.a 2024-03-29T11:12:10.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.921+0000 7f8ffa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:10.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.921+0000 7f8ffa7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:12:10.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.921+0000 7f8ffa7fc640 1 -- 172.21.15.18:0/1853784838 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9004071410 con 0x7f90040ad6c0 2024-03-29T11:12:10.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.921+0000 7f8ffa7fc640 10 monclient: _finish_auth 0 2024-03-29T11:12:10.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.921+0000 7f8ffa7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:12:10.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.921+0000 7f8ffa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:10.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.922+0000 7f900bee2640 5 monclient: authenticate success, global_id 4329 2024-03-29T11:12:10.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.922+0000 7f900bee2640 10 monclient: _renew_subs 2024-03-29T11:12:10.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.922+0000 7f900bee2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:12:10.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.922+0000 7f900bee2640 1 -- 172.21.15.18:0/1853784838 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9004159d90 con 0x7f90040ad6c0 2024-03-29T11:12:10.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.922+0000 7f900bee2640 10 client.4329.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:10.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.922+0000 7f900bee2640 10 monclient: _renew_subs 2024-03-29T11:12:10.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.922+0000 7f900bee2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:12:10.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.922+0000 7f900bee2640 1 -- 172.21.15.18:0/1853784838 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9004072600 con 0x7f90040ad6c0 2024-03-29T11:12:10.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.922+0000 7f900bee2640 1 librados: init done 2024-03-29T11:12:10.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.923+0000 7f8ffa7fc640 1 -- 172.21.15.18:0/1853784838 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9000001ce0 con 0x7f90040ad6c0 2024-03-29T11:12:10.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.923+0000 7f8ffa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:10.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.923+0000 7f8ffbfff640 4 set_mon_vals no callback set 2024-03-29T11:12:10.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.923+0000 7f900bee2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:10.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.924+0000 7f8ffa7fc640 1 -- 172.21.15.18:0/1853784838 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f90000019e0 con 0x7f90040ad6c0 2024-03-29T11:12:10.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.924+0000 7f8ffa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:10.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.924+0000 7f8ffa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:10.909 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.924+0000 7f8ffa7fc640 10 monclient: dump: 2024-03-29T11:12:10.909 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:12:10.909 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:10.909 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:10.909 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:10.910 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:10.910 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:10.910 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:12:10.910 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:12:10.910 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:12:10.910 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:10.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.925+0000 7f8ffa7fc640 1 -- 172.21.15.18:0/1853784838 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f9000003980 con 0x7f90040ad6c0 2024-03-29T11:12:10.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.925+0000 7f900bee2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:10.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.925+0000 7f900bee2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:12:10.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.925+0000 7f900bee2640 1 -- 172.21.15.18:0/1853784838 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9004075f10 con 0x7f90040ad6c0 2024-03-29T11:12:10.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.926+0000 7f900947d640 10 monclient: get_auth_request con 0x7f8fe806df90 auth_method 0 2024-03-29T11:12:10.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.926+0000 7f8ffa7fc640 1 -- 172.21.15.18:0/1853784838 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f90000848f0 con 0x7f90040ad6c0 2024-03-29T11:12:10.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.926+0000 7f8ffa7fc640 10 client.4329.objecter ms_dispatch 0x7f9004000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:12:10.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.926+0000 7f8ffa7fc640 3 client.4329.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:12:10.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.926+0000 7f8ffa7fc640 3 client.4329.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:12:10.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.926+0000 7f8ffa7fc640 20 client.4329.objecter dump_active .. 0 homeless 2024-03-29T11:12:10.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.929+0000 7f8ffa7fc640 10 client.4329.objecter ms_handle_connect 0x7f8fe806df90 2024-03-29T11:12:10.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.934+0000 7f8ffa7fc640 1 -- 172.21.15.18:0/1853784838 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f90000501d0 con 0x7f90040ad6c0 2024-03-29T11:12:10.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.934+0000 7f8ffa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:10.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:10.934+0000 7f8ffa7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:11.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:11.264+0000 7f900bee2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:11.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:11.264+0000 7f900bee2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:11.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:11.264+0000 7f900bee2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:12:11.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:11.264+0000 7f900bee2640 1 -- 172.21.15.18:0/1853784838 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90040719c0 con 0x7f90040ad6c0 2024-03-29T11:12:11.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:11.266+0000 7f8ffa7fc640 1 -- 172.21.15.18:0/1853784838 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f9000052100 con 0x7f90040ad6c0 2024-03-29T11:12:11.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:11.266+0000 7f8ffa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:11.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:11.266+0000 7f8ffa7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:11.249 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:12:11.250 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:11.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:11.270+0000 7f900bee2640 10 librados: watch_flush enter 2024-03-29T11:12:11.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:11.270+0000 7f900bee2640 10 librados: watch_flush exit 2024-03-29T11:12:11.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:11.271+0000 7f900bee2640 20 client.4329.objecter shutdown clearing up homeless session... 2024-03-29T11:12:11.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:11.271+0000 7f900bee2640 10 client.4329.objecter successfully canceled tick 2024-03-29T11:12:11.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:11.271+0000 7f900bee2640 1 -- 172.21.15.18:0/1853784838 >> v1:172.21.15.70:6816/3101871562 conn(0x7f8fe806df90 legacy=0x7f8fe8070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:11.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:11.271+0000 7f900bee2640 10 monclient: shutdown 2024-03-29T11:12:11.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:11.271+0000 7f900bee2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:11.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:11.271+0000 7f900bee2640 1 -- 172.21.15.18:0/1853784838 >> v1:172.21.15.18:6789/0 conn(0x7f90040ad6c0 legacy=0x7f900406f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:11.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:11.271+0000 7f900bee2640 1 -- 172.21.15.18:0/1853784838 shutdown_connections 2024-03-29T11:12:11.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:11.271+0000 7f900bee2640 1 -- 172.21.15.18:0/1853784838 >> 172.21.15.18:0/1853784838 conn(0x7f90040b35e0 msgr2=0x7f90040a94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:11.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:11.272+0000 7f900bee2640 1 -- 172.21.15.18:0/1853784838 shutdown_connections 2024-03-29T11:12:11.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:11.272+0000 7f900bee2640 1 -- 172.21.15.18:0/1853784838 wait complete. 2024-03-29T11:12:11.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:11.272+0000 7f900bee2640 1 librados: shutdown 2024-03-29T11:12:11.271 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:12:18.079 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:18.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.253+0000 7fde91015640 10 monclient: get_monmap_and_config 2024-03-29T11:12:18.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.253+0000 7fde91015640 10 monclient: build_initial_monmap 2024-03-29T11:12:18.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.253+0000 7fde91015640 10 monclient: monmap: 2024-03-29T11:12:18.232 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:12:18.232 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:18.233 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:12:18.254172+0000 2024-03-29T11:12:18.233 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:12:18.254172+0000 2024-03-29T11:12:18.233 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:18.233 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:18.233 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:12:18.233 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:12:18.233 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:12:18.233 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:18.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.255+0000 7fde91015640 1 Processor -- start 2024-03-29T11:12:18.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.256+0000 7fde91015640 1 -- start start 2024-03-29T11:12:18.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.256+0000 7fde91015640 10 monclient: init 2024-03-29T11:12:18.237 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.257+0000 7fde91015640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:18.237 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.257+0000 7fde91015640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:12:18.237 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.257+0000 7fde91015640 10 monclient(hunting): picked mon.noname-c con 0x7fde8c0b4100 addr v1:172.21.15.70:6789/0 2024-03-29T11:12:18.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.257+0000 7fde91015640 10 monclient(hunting): picked mon.noname-a con 0x7fde8c13ee90 addr v1:172.21.15.18:6789/0 2024-03-29T11:12:18.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.260+0000 7fde91015640 10 monclient(hunting): picked mon.noname-b con 0x7fde8c0af900 addr v1:172.21.15.18:6790/0 2024-03-29T11:12:18.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.260+0000 7fde91015640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde8c0afd10 con 0x7fde8c13ee90 2024-03-29T11:12:18.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.260+0000 7fde91015640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde8c0af6f0 con 0x7fde8c0b4100 2024-03-29T11:12:18.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.260+0000 7fde91015640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde8c0aaf60 con 0x7fde8c0af900 2024-03-29T11:12:18.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.260+0000 7fde91015640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:18.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.260+0000 7fde91015640 10 monclient(hunting): authenticate will time out at 964.532022s 2024-03-29T11:12:18.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.260+0000 7fde8b59e640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fde8c0af900 0x7fde8c13b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:47663/0 (socket says 172.21.15.18:47663) 2024-03-29T11:12:18.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.260+0000 7fde8b59e640 1 -- 172.21.15.18:0/2618043219 learned_addr learned my addr 172.21.15.18:0/2618043219 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:12:18.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.261+0000 7fde89d9b640 1 -- 172.21.15.18:0/2618043219 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1139586503 0 0) 0x7fde8c0aaf60 con 0x7fde8c0af900 2024-03-29T11:12:18.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.261+0000 7fde89d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:12:18.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.261+0000 7fde89d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:18.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.261+0000 7fde89d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:18.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.261+0000 7fde89d9b640 10 monclient(hunting): my global_id is 4361 2024-03-29T11:12:18.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.261+0000 7fde89d9b640 1 -- 172.21.15.18:0/2618043219 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde74003520 con 0x7fde8c0af900 2024-03-29T11:12:18.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.261+0000 7fde89d9b640 1 -- 172.21.15.18:0/2618043219 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 261912308 0 0) 0x7fde74003520 con 0x7fde8c0af900 2024-03-29T11:12:18.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.261+0000 7fde89d9b640 1 -- 172.21.15.18:0/2618043219 >> v1:172.21.15.70:6789/0 conn(0x7fde8c0b4100 legacy=0x7fde8c0a9850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:12:18.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.263+0000 7fde89d9b640 1 -- 172.21.15.18:0/2618043219 >> v1:172.21.15.18:6789/0 conn(0x7fde8c13ee90 legacy=0x7fde8c0adfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:12:18.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.264+0000 7fde89d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:12:18.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.264+0000 7fde89d9b640 1 monclient: found mon.noname-b 2024-03-29T11:12:18.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.264+0000 7fde89d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:18.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.264+0000 7fde89d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:12:18.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.264+0000 7fde89d9b640 1 -- 172.21.15.18:0/2618043219 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde8c13e970 con 0x7fde8c0af900 2024-03-29T11:12:18.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.265+0000 7fde89d9b640 10 monclient: _finish_auth 0 2024-03-29T11:12:18.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.265+0000 7fde89d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:12:18.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.265+0000 7fde89d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:18.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.265+0000 7fde89d9b640 1 -- 172.21.15.18:0/2618043219 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fde84001a10 con 0x7fde8c0af900 2024-03-29T11:12:18.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.265+0000 7fde89d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:18.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.265+0000 7fde89d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:18.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.265+0000 7fde89d9b640 10 monclient: dump: 2024-03-29T11:12:18.246 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:12:18.246 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:18.246 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:18.246 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:18.246 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:18.246 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:18.246 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:12:18.246 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:12:18.246 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:12:18.247 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:18.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.265+0000 7fde89d9b640 1 -- 172.21.15.18:0/2618043219 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde84001ca0 con 0x7fde8c0af900 2024-03-29T11:12:18.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.265+0000 7fde89d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:18.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.265+0000 7fde89d9b640 1 -- 172.21.15.18:0/2618043219 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fde84002030 con 0x7fde8c0af900 2024-03-29T11:12:18.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.265+0000 7fde89d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:18.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.265+0000 7fde89d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:18.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.265+0000 7fde89d9b640 10 monclient: dump: 2024-03-29T11:12:18.247 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:12:18.247 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:18.247 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:18.247 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:18.248 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:18.248 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:18.248 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:12:18.248 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:12:18.248 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:12:18.248 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:18.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.266+0000 7fde91015640 5 monclient: authenticate success, global_id 4361 2024-03-29T11:12:18.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.266+0000 7fde91015640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:18.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.266+0000 7fde91015640 4 set_mon_vals no callback set 2024-03-29T11:12:18.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.267+0000 7fde91015640 10 monclient: shutdown 2024-03-29T11:12:18.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.267+0000 7fde91015640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:18.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.267+0000 7fde91015640 1 -- 172.21.15.18:0/2618043219 >> v1:172.21.15.18:6790/0 conn(0x7fde8c0af900 legacy=0x7fde8c13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:18.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.268+0000 7fde91015640 1 -- 172.21.15.18:0/2618043219 shutdown_connections 2024-03-29T11:12:18.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.268+0000 7fde91015640 1 -- 172.21.15.18:0/2618043219 >> 172.21.15.18:0/2618043219 conn(0x7fde8c0b35e0 msgr2=0x7fde8c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:18.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.269+0000 7fde91015640 1 -- 172.21.15.18:0/2618043219 shutdown_connections 2024-03-29T11:12:18.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.269+0000 7fde91015640 1 -- 172.21.15.18:0/2618043219 wait complete. 2024-03-29T11:12:18.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.270+0000 7fde91015640 10 monclient: build_initial_monmap 2024-03-29T11:12:18.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.270+0000 7fde91015640 10 monclient: monmap: 2024-03-29T11:12:18.251 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:12:18.251 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:18.251 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:12:18.251 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:12:18.251 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:18.251 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:18.251 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:12:18.251 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:12:18.252 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:12:18.252 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:18.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.271+0000 7fde91015640 1 librados: starting msgr at 2024-03-29T11:12:18.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.271+0000 7fde91015640 1 librados: starting objecter 2024-03-29T11:12:18.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.271+0000 7fde91015640 1 Processor -- start 2024-03-29T11:12:18.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.271+0000 7fde91015640 1 -- start start 2024-03-29T11:12:18.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.271+0000 7fde91015640 1 librados: setting wanted keys 2024-03-29T11:12:18.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.271+0000 7fde91015640 1 librados: calling monclient init 2024-03-29T11:12:18.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.272+0000 7fde91015640 10 monclient: init 2024-03-29T11:12:18.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.272+0000 7fde91015640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:18.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.273+0000 7fde91015640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:12:18.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.273+0000 7fde91015640 10 monclient(hunting): picked mon.noname-b con 0x7fde8c0af900 addr v1:172.21.15.70:6789/0 2024-03-29T11:12:18.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.273+0000 7fde91015640 10 monclient(hunting): picked mon.noname-c con 0x7fde8c0b4100 addr v1:172.21.15.18:6790/0 2024-03-29T11:12:18.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.273+0000 7fde91015640 10 monclient(hunting): picked mon.noname-a con 0x7fde8c13ee90 addr v1:172.21.15.18:6789/0 2024-03-29T11:12:18.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.274+0000 7fde91015640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde8c06f920 con 0x7fde8c13ee90 2024-03-29T11:12:18.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.274+0000 7fde91015640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde8c141e60 con 0x7fde8c0af900 2024-03-29T11:12:18.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.274+0000 7fde91015640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde8c1d8dc0 con 0x7fde8c0b4100 2024-03-29T11:12:18.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.274+0000 7fde91015640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:18.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.274+0000 7fde91015640 10 monclient(hunting): authenticate will time out at 964.546004s 2024-03-29T11:12:18.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.279+0000 7fde8b59e640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fde8c13ee90 0x7fde8c1e09f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:38675/0 (socket says 172.21.15.18:38675) 2024-03-29T11:12:18.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.279+0000 7fde8b59e640 1 -- 172.21.15.18:0/174426763 learned_addr learned my addr 172.21.15.18:0/174426763 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:12:18.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.279+0000 7fde737fe640 10 client.?.objecter ms_handle_connect 0x7fde8c13ee90 2024-03-29T11:12:18.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.279+0000 7fde737fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:18.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.280+0000 7fde737fe640 10 client.?.objecter ms_handle_connect 0x7fde8c0af900 2024-03-29T11:12:18.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.280+0000 7fde737fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:18.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.281+0000 7fde737fe640 10 client.?.objecter ms_handle_connect 0x7fde8c0b4100 2024-03-29T11:12:18.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.281+0000 7fde737fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:18.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.281+0000 7fde737fe640 1 -- 172.21.15.18:0/174426763 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2870197573 0 0) 0x7fde8c06f920 con 0x7fde8c13ee90 2024-03-29T11:12:18.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.281+0000 7fde737fe640 10 monclient(hunting): _negotiate 2024-03-29T11:12:18.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.281+0000 7fde737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:18.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.281+0000 7fde737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:18.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.281+0000 7fde737fe640 10 monclient(hunting): my global_id is 4332 2024-03-29T11:12:18.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.281+0000 7fde737fe640 1 -- 172.21.15.18:0/174426763 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde78004eb0 con 0x7fde8c13ee90 2024-03-29T11:12:18.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.283+0000 7fde737fe640 1 -- 172.21.15.18:0/174426763 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3027947720 0 0) 0x7fde78004eb0 con 0x7fde8c13ee90 2024-03-29T11:12:18.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.283+0000 7fde737fe640 1 -- 172.21.15.18:0/174426763 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde78005300 con 0x7fde8c13ee90 2024-03-29T11:12:18.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.283+0000 7fde737fe640 1 -- 172.21.15.18:0/174426763 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 309181649 0 0) 0x7fde8c1d8dc0 con 0x7fde8c0b4100 2024-03-29T11:12:18.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.283+0000 7fde737fe640 10 monclient(hunting): _negotiate 2024-03-29T11:12:18.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.283+0000 7fde737fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:18.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.283+0000 7fde737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:18.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.283+0000 7fde737fe640 10 monclient(hunting): my global_id is 4364 2024-03-29T11:12:18.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.283+0000 7fde737fe640 1 -- 172.21.15.18:0/174426763 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde78004eb0 con 0x7fde8c0b4100 2024-03-29T11:12:18.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.283+0000 7fde737fe640 1 -- 172.21.15.18:0/174426763 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fde840018d0 con 0x7fde8c13ee90 2024-03-29T11:12:18.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.283+0000 7fde737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:18.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.283+0000 7fde737fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:12:18.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.283+0000 7fde737fe640 10 monclient(hunting): dump: 2024-03-29T11:12:18.263 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:12:18.263 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:18.263 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:18.263 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:18.263 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:18.264 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:18.264 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:12:18.264 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:12:18.264 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:12:18.264 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:18.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.283+0000 7fde737fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:18.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.283+0000 7fde737fe640 1 -- 172.21.15.18:0/174426763 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4213006695 0 0) 0x7fde78005300 con 0x7fde8c13ee90 2024-03-29T11:12:18.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.283+0000 7fde737fe640 1 -- 172.21.15.18:0/174426763 >> v1:172.21.15.18:6790/0 conn(0x7fde8c0b4100 legacy=0x7fde8c1326f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:18.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.283+0000 7fde737fe640 1 -- 172.21.15.18:0/174426763 >> v1:172.21.15.70:6789/0 conn(0x7fde8c0af900 legacy=0x7fde8c1d86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:18.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.283+0000 7fde737fe640 10 monclient: _finish_hunting 0 2024-03-29T11:12:18.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.283+0000 7fde737fe640 1 monclient: found mon.a 2024-03-29T11:12:18.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.283+0000 7fde737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:18.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.283+0000 7fde737fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:12:18.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.283+0000 7fde737fe640 1 -- 172.21.15.18:0/174426763 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde8c0724c0 con 0x7fde8c13ee90 2024-03-29T11:12:18.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.283+0000 7fde737fe640 10 monclient: _finish_auth 0 2024-03-29T11:12:18.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.283+0000 7fde737fe640 10 monclient: _check_auth_tickets 2024-03-29T11:12:18.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.283+0000 7fde737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:18.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.283+0000 7fde91015640 5 monclient: authenticate success, global_id 4332 2024-03-29T11:12:18.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.283+0000 7fde91015640 10 monclient: _renew_subs 2024-03-29T11:12:18.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.283+0000 7fde91015640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:12:18.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.283+0000 7fde91015640 1 -- 172.21.15.18:0/174426763 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde8c06e7a0 con 0x7fde8c13ee90 2024-03-29T11:12:18.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.283+0000 7fde91015640 10 client.4332.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:18.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.283+0000 7fde91015640 10 monclient: _renew_subs 2024-03-29T11:12:18.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.283+0000 7fde91015640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:12:18.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.283+0000 7fde91015640 1 -- 172.21.15.18:0/174426763 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fde8c142560 con 0x7fde8c13ee90 2024-03-29T11:12:18.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.283+0000 7fde91015640 1 librados: init done 2024-03-29T11:12:18.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.284+0000 7fde737fe640 1 -- 172.21.15.18:0/174426763 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde84004f70 con 0x7fde8c13ee90 2024-03-29T11:12:18.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.284+0000 7fde737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:18.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.284+0000 7fde737fe640 1 -- 172.21.15.18:0/174426763 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fde84005280 con 0x7fde8c13ee90 2024-03-29T11:12:18.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.284+0000 7fde737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:18.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.284+0000 7fde737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:12:18.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.284+0000 7fde737fe640 10 monclient: dump: 2024-03-29T11:12:18.266 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:12:18.266 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:18.266 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:18.266 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:18.266 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:18.266 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:18.266 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:12:18.266 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:12:18.266 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:12:18.266 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:18.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.285+0000 7fde91015640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:18.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.285+0000 7fde91015640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:18.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.285+0000 7fde91015640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:12:18.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.285+0000 7fde8959a640 4 set_mon_vals no callback set 2024-03-29T11:12:18.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.285+0000 7fde91015640 1 -- 172.21.15.18:0/174426763 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde8c141cc0 con 0x7fde8c13ee90 2024-03-29T11:12:18.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.288+0000 7fde737fe640 1 -- 172.21.15.18:0/174426763 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7fde8401b250 con 0x7fde8c13ee90 2024-03-29T11:12:18.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.290+0000 7fde8ad9d640 10 monclient: get_auth_request con 0x7fde7806dfa0 auth_method 0 2024-03-29T11:12:18.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.290+0000 7fde737fe640 1 -- 172.21.15.18:0/174426763 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7fde84054c00 con 0x7fde8c13ee90 2024-03-29T11:12:18.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.290+0000 7fde737fe640 10 client.4332.objecter ms_dispatch 0x7fde8c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:12:18.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.290+0000 7fde737fe640 3 client.4332.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:12:18.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.290+0000 7fde737fe640 3 client.4332.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:12:18.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.290+0000 7fde737fe640 20 client.4332.objecter dump_active .. 0 homeless 2024-03-29T11:12:18.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.290+0000 7fde737fe640 10 client.4332.objecter ms_handle_connect 0x7fde7806dfa0 2024-03-29T11:12:18.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.295+0000 7fde737fe640 1 -- 172.21.15.18:0/174426763 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fde8c141cc0 con 0x7fde8c13ee90 2024-03-29T11:12:18.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.295+0000 7fde737fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:18.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.295+0000 7fde737fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:18.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.656+0000 7fde91015640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:18.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.656+0000 7fde91015640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:18.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.656+0000 7fde91015640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:12:18.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.656+0000 7fde91015640 1 -- 172.21.15.18:0/174426763 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fde8c0aaf00 con 0x7fde8c13ee90 2024-03-29T11:12:18.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.658+0000 7fde737fe640 1 -- 172.21.15.18:0/174426763 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7fde8c141cc0 con 0x7fde8c13ee90 2024-03-29T11:12:18.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.658+0000 7fde737fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:18.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.658+0000 7fde737fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:18.636 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:12:18.637 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:18.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.660+0000 7fde91015640 10 librados: watch_flush enter 2024-03-29T11:12:18.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.660+0000 7fde91015640 10 librados: watch_flush exit 2024-03-29T11:12:18.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.660+0000 7fde91015640 20 client.4332.objecter shutdown clearing up homeless session... 2024-03-29T11:12:18.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.660+0000 7fde91015640 10 client.4332.objecter successfully canceled tick 2024-03-29T11:12:18.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.660+0000 7fde91015640 1 -- 172.21.15.18:0/174426763 >> v1:172.21.15.70:6816/3101871562 conn(0x7fde7806dfa0 legacy=0x7fde78070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:18.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.660+0000 7fde91015640 10 monclient: shutdown 2024-03-29T11:12:18.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.660+0000 7fde91015640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:18.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.660+0000 7fde91015640 1 -- 172.21.15.18:0/174426763 >> v1:172.21.15.18:6789/0 conn(0x7fde8c13ee90 legacy=0x7fde8c1e09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:18.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.661+0000 7fde91015640 1 -- 172.21.15.18:0/174426763 shutdown_connections 2024-03-29T11:12:18.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.661+0000 7fde91015640 1 -- 172.21.15.18:0/174426763 >> 172.21.15.18:0/174426763 conn(0x7fde8c0b35e0 msgr2=0x7fde8c1428b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:18.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.661+0000 7fde91015640 1 -- 172.21.15.18:0/174426763 shutdown_connections 2024-03-29T11:12:18.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.661+0000 7fde91015640 1 -- 172.21.15.18:0/174426763 wait complete. 2024-03-29T11:12:18.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:18.661+0000 7fde91015640 1 librados: shutdown 2024-03-29T11:12:18.653 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:12:25.461 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:25.612 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.633+0000 7fcc7a7fb640 10 monclient: get_monmap_and_config 2024-03-29T11:12:25.612 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.633+0000 7fcc7a7fb640 10 monclient: build_initial_monmap 2024-03-29T11:12:25.613 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.633+0000 7fcc7a7fb640 10 monclient: monmap: 2024-03-29T11:12:25.613 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:12:25.613 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:25.613 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:12:25.635255+0000 2024-03-29T11:12:25.613 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:12:25.635255+0000 2024-03-29T11:12:25.613 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:25.613 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:25.613 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:12:25.613 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:12:25.613 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:12:25.613 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:25.614 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.634+0000 7fcc7a7fb640 1 Processor -- start 2024-03-29T11:12:25.614 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.634+0000 7fcc7a7fb640 1 -- start start 2024-03-29T11:12:25.614 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.634+0000 7fcc7a7fb640 10 monclient: init 2024-03-29T11:12:25.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.635+0000 7fcc7a7fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:25.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.635+0000 7fcc7a7fb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:12:25.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.635+0000 7fcc7a7fb640 10 monclient(hunting): picked mon.noname-b con 0x7fcc740b4330 addr v1:172.21.15.18:6790/0 2024-03-29T11:12:25.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.635+0000 7fcc7a7fb640 10 monclient(hunting): picked mon.noname-a con 0x7fcc741470b0 addr v1:172.21.15.18:6789/0 2024-03-29T11:12:25.617 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.635+0000 7fcc7a7fb640 10 monclient(hunting): picked mon.noname-c con 0x7fcc741439b0 addr v1:172.21.15.70:6789/0 2024-03-29T11:12:25.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.635+0000 7fcc7a7fb640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc7406f2a0 con 0x7fcc741470b0 2024-03-29T11:12:25.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.635+0000 7fcc7a7fb640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc74146b90 con 0x7fcc741439b0 2024-03-29T11:12:25.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.635+0000 7fcc7a7fb640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc740b2ca0 con 0x7fcc740b4330 2024-03-29T11:12:25.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.635+0000 7fcc7a7fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:25.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.635+0000 7fcc7a7fb640 10 monclient(hunting): authenticate will time out at 971.908255s 2024-03-29T11:12:25.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.635+0000 7fcc73fff640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fcc740b4330 0x7fcc740b4710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:51594/0 (socket says 172.21.15.18:51594) 2024-03-29T11:12:25.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.635+0000 7fcc73fff640 1 -- 172.21.15.18:0/3109812836 learned_addr learned my addr 172.21.15.18:0/3109812836 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:12:25.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.636+0000 7fcc72ffd640 1 -- 172.21.15.18:0/3109812836 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 249427395 0 0) 0x7fcc740b2ca0 con 0x7fcc740b4330 2024-03-29T11:12:25.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.636+0000 7fcc72ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:25.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.636+0000 7fcc72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:25.618 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.636+0000 7fcc72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:25.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.636+0000 7fcc72ffd640 10 monclient(hunting): my global_id is 4367 2024-03-29T11:12:25.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.636+0000 7fcc72ffd640 1 -- 172.21.15.18:0/3109812836 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc5c003520 con 0x7fcc740b4330 2024-03-29T11:12:25.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.637+0000 7fcc72ffd640 1 -- 172.21.15.18:0/3109812836 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1621007194 0 0) 0x7fcc7406f2a0 con 0x7fcc741470b0 2024-03-29T11:12:25.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.637+0000 7fcc72ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:25.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.637+0000 7fcc72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:25.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.637+0000 7fcc72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:25.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.637+0000 7fcc72ffd640 10 monclient(hunting): my global_id is 4335 2024-03-29T11:12:25.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.637+0000 7fcc72ffd640 1 -- 172.21.15.18:0/3109812836 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc740b2ca0 con 0x7fcc741470b0 2024-03-29T11:12:25.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.637+0000 7fcc72ffd640 1 -- 172.21.15.18:0/3109812836 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3200490998 0 0) 0x7fcc5c003520 con 0x7fcc740b4330 2024-03-29T11:12:25.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.637+0000 7fcc72ffd640 1 -- 172.21.15.18:0/3109812836 >> v1:172.21.15.70:6789/0 conn(0x7fcc741439b0 legacy=0x7fcc74151d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:12:25.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.640+0000 7fcc72ffd640 1 -- 172.21.15.18:0/3109812836 >> v1:172.21.15.18:6789/0 conn(0x7fcc741470b0 legacy=0x7fcc740b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:25.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.640+0000 7fcc72ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:12:25.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.640+0000 7fcc72ffd640 1 monclient: found mon.noname-b 2024-03-29T11:12:25.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.640+0000 7fcc72ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:25.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.640+0000 7fcc72ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:12:25.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.640+0000 7fcc72ffd640 1 -- 172.21.15.18:0/3109812836 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc7406e0c0 con 0x7fcc740b4330 2024-03-29T11:12:25.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.640+0000 7fcc72ffd640 10 monclient: _finish_auth 0 2024-03-29T11:12:25.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.640+0000 7fcc72ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:12:25.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.640+0000 7fcc72ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:25.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.640+0000 7fcc7a7fb640 5 monclient: authenticate success, global_id 4367 2024-03-29T11:12:25.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.640+0000 7fcc7a7fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:25.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.640+0000 7fcc72ffd640 1 -- 172.21.15.18:0/3109812836 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fcc64001610 con 0x7fcc740b4330 2024-03-29T11:12:25.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.640+0000 7fcc72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:25.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.640+0000 7fcc72ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:25.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.640+0000 7fcc72ffd640 10 monclient: dump: 2024-03-29T11:12:25.623 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:12:25.623 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:25.623 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:25.623 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:25.623 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:25.623 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:25.624 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:12:25.624 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:12:25.624 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:12:25.624 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:25.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.640+0000 7fcc7a7fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:25.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.640+0000 7fcc72ffd640 1 -- 172.21.15.18:0/3109812836 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc64001830 con 0x7fcc740b4330 2024-03-29T11:12:25.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.640+0000 7fcc72ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:25.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.640+0000 7fcc7a7fb640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:25.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.640+0000 7fcc7a7fb640 4 set_mon_vals no callback set 2024-03-29T11:12:25.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.640+0000 7fcc72ffd640 1 -- 172.21.15.18:0/3109812836 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fcc64001bc0 con 0x7fcc740b4330 2024-03-29T11:12:25.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.640+0000 7fcc72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:25.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.640+0000 7fcc72ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:25.624 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.640+0000 7fcc72ffd640 10 monclient: dump: 2024-03-29T11:12:25.625 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:12:25.625 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:25.625 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:25.625 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:25.625 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:25.625 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:25.625 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:12:25.625 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:12:25.625 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:12:25.625 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:25.625 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.641+0000 7fcc7a7fb640 10 monclient: shutdown 2024-03-29T11:12:25.625 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.641+0000 7fcc7a7fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:25.625 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.641+0000 7fcc7a7fb640 1 -- 172.21.15.18:0/3109812836 >> v1:172.21.15.18:6790/0 conn(0x7fcc740b4330 legacy=0x7fcc740b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:25.625 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.641+0000 7fcc7a7fb640 1 -- 172.21.15.18:0/3109812836 shutdown_connections 2024-03-29T11:12:25.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.641+0000 7fcc7a7fb640 1 -- 172.21.15.18:0/3109812836 >> 172.21.15.18:0/3109812836 conn(0x7fcc74066a00 msgr2=0x7fcc74066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:25.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.641+0000 7fcc7a7fb640 1 -- 172.21.15.18:0/3109812836 shutdown_connections 2024-03-29T11:12:25.626 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.642+0000 7fcc7a7fb640 1 -- 172.21.15.18:0/3109812836 wait complete. 2024-03-29T11:12:25.627 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.642+0000 7fcc7a7fb640 10 monclient: build_initial_monmap 2024-03-29T11:12:25.627 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.642+0000 7fcc7a7fb640 10 monclient: monmap: 2024-03-29T11:12:25.627 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:12:25.627 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:25.627 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:12:25.627 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:12:25.627 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:25.627 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:25.627 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:12:25.628 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:12:25.628 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:12:25.628 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:25.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.643+0000 7fcc7a7fb640 1 librados: starting msgr at 2024-03-29T11:12:25.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.643+0000 7fcc7a7fb640 1 librados: starting objecter 2024-03-29T11:12:25.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.643+0000 7fcc7a7fb640 1 Processor -- start 2024-03-29T11:12:25.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.643+0000 7fcc7a7fb640 1 -- start start 2024-03-29T11:12:25.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.643+0000 7fcc7a7fb640 1 librados: setting wanted keys 2024-03-29T11:12:25.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.643+0000 7fcc7a7fb640 1 librados: calling monclient init 2024-03-29T11:12:25.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.643+0000 7fcc7a7fb640 10 monclient: init 2024-03-29T11:12:25.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.644+0000 7fcc7a7fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:25.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.644+0000 7fcc7a7fb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:12:25.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.644+0000 7fcc7a7fb640 10 monclient(hunting): picked mon.noname-a con 0x7fcc740b4330 addr v1:172.21.15.18:6789/0 2024-03-29T11:12:25.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.644+0000 7fcc7a7fb640 10 monclient(hunting): picked mon.noname-b con 0x7fcc741439b0 addr v1:172.21.15.70:6789/0 2024-03-29T11:12:25.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.644+0000 7fcc7a7fb640 10 monclient(hunting): picked mon.noname-c con 0x7fcc741470b0 addr v1:172.21.15.18:6790/0 2024-03-29T11:12:25.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.644+0000 7fcc7a7fb640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc741214c0 con 0x7fcc740b4330 2024-03-29T11:12:25.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.644+0000 7fcc7a7fb640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc741460b0 con 0x7fcc741439b0 2024-03-29T11:12:25.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.644+0000 7fcc7a7fb640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc74146390 con 0x7fcc741470b0 2024-03-29T11:12:25.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.644+0000 7fcc7a7fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:25.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.644+0000 7fcc7a7fb640 10 monclient(hunting): authenticate will time out at 971.917304s 2024-03-29T11:12:25.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.645+0000 7fcc78d98640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fcc741470b0 0x7fcc741e6bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:51770/0 (socket says 172.21.15.18:51770) 2024-03-29T11:12:25.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.645+0000 7fcc78d98640 1 -- 172.21.15.18:0/1101905440 learned_addr learned my addr 172.21.15.18:0/1101905440 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:12:25.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.646+0000 7fcc70ff9640 10 client.?.objecter ms_handle_connect 0x7fcc741470b0 2024-03-29T11:12:25.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.646+0000 7fcc70ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:25.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.646+0000 7fcc70ff9640 10 client.?.objecter ms_handle_connect 0x7fcc740b4330 2024-03-29T11:12:25.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.646+0000 7fcc70ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:25.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.648+0000 7fcc70ff9640 10 client.?.objecter ms_handle_connect 0x7fcc741439b0 2024-03-29T11:12:25.630 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.648+0000 7fcc70ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:25.630 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.648+0000 7fcc70ff9640 1 -- 172.21.15.18:0/1101905440 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3508246591 0 0) 0x7fcc74146390 con 0x7fcc741470b0 2024-03-29T11:12:25.630 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.648+0000 7fcc70ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:12:25.630 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.648+0000 7fcc70ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:25.630 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.648+0000 7fcc70ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:25.630 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.648+0000 7fcc70ff9640 10 monclient(hunting): my global_id is 4370 2024-03-29T11:12:25.630 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.648+0000 7fcc70ff9640 1 -- 172.21.15.18:0/1101905440 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc60004eb0 con 0x7fcc741470b0 2024-03-29T11:12:25.630 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.648+0000 7fcc70ff9640 1 -- 172.21.15.18:0/1101905440 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3412944629 0 0) 0x7fcc741460b0 con 0x7fcc741439b0 2024-03-29T11:12:25.630 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.648+0000 7fcc70ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:12:25.630 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.648+0000 7fcc70ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:25.630 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.648+0000 7fcc70ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:25.630 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.648+0000 7fcc70ff9640 10 monclient(hunting): my global_id is 4342 2024-03-29T11:12:25.630 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.648+0000 7fcc70ff9640 1 -- 172.21.15.18:0/1101905440 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc74146390 con 0x7fcc741439b0 2024-03-29T11:12:25.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.649+0000 7fcc70ff9640 1 -- 172.21.15.18:0/1101905440 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4281341422 0 0) 0x7fcc74146390 con 0x7fcc741439b0 2024-03-29T11:12:25.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.649+0000 7fcc70ff9640 1 -- 172.21.15.18:0/1101905440 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc60005490 con 0x7fcc741439b0 2024-03-29T11:12:25.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.649+0000 7fcc70ff9640 1 -- 172.21.15.18:0/1101905440 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3529142400 0 0) 0x7fcc60004eb0 con 0x7fcc741470b0 2024-03-29T11:12:25.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.649+0000 7fcc70ff9640 1 -- 172.21.15.18:0/1101905440 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc74146390 con 0x7fcc741470b0 2024-03-29T11:12:25.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.649+0000 7fcc70ff9640 1 -- 172.21.15.18:0/1101905440 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fcc6c001590 con 0x7fcc741439b0 2024-03-29T11:12:25.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.649+0000 7fcc70ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:25.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.649+0000 7fcc70ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:25.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.649+0000 7fcc70ff9640 10 monclient(hunting): dump: 2024-03-29T11:12:25.636 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:12:25.636 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:25.636 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:25.636 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:25.636 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:25.636 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:25.636 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:12:25.636 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:12:25.636 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:12:25.637 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:25.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.649+0000 7fcc70ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:25.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.649+0000 7fcc70ff9640 1 -- 172.21.15.18:0/1101905440 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fcc680016d0 con 0x7fcc741470b0 2024-03-29T11:12:25.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.649+0000 7fcc70ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:25.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.649+0000 7fcc70ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:25.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.649+0000 7fcc70ff9640 10 monclient(hunting): dump: 2024-03-29T11:12:25.637 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:12:25.637 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:25.637 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:25.637 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:25.637 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:25.637 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:25.637 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:12:25.637 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:12:25.638 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:12:25.638 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:25.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.649+0000 7fcc70ff9640 1 -- 172.21.15.18:0/1101905440 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3719902863 0 0) 0x7fcc741214c0 con 0x7fcc740b4330 2024-03-29T11:12:25.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.649+0000 7fcc70ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:12:25.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.649+0000 7fcc70ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:25.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.649+0000 7fcc70ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:25.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.649+0000 7fcc70ff9640 10 monclient(hunting): my global_id is 4338 2024-03-29T11:12:25.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.649+0000 7fcc70ff9640 1 -- 172.21.15.18:0/1101905440 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc60004eb0 con 0x7fcc740b4330 2024-03-29T11:12:25.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.650+0000 7fcc70ff9640 1 -- 172.21.15.18:0/1101905440 <== mon.1 v1:172.21.15.70:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2223044768 0 0) 0x7fcc60005490 con 0x7fcc741439b0 2024-03-29T11:12:25.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.650+0000 7fcc70ff9640 1 -- 172.21.15.18:0/1101905440 >> v1:172.21.15.18:6790/0 conn(0x7fcc741470b0 legacy=0x7fcc741e6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:25.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.650+0000 7fcc70ff9640 1 -- 172.21.15.18:0/1101905440 >> v1:172.21.15.18:6789/0 conn(0x7fcc740b4330 legacy=0x7fcc7411f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:25.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.650+0000 7fcc70ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:12:25.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.650+0000 7fcc70ff9640 1 monclient: found mon.b 2024-03-29T11:12:25.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.650+0000 7fcc70ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:25.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.650+0000 7fcc70ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:12:25.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.650+0000 7fcc70ff9640 1 -- 172.21.15.18:0/1101905440 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc741db380 con 0x7fcc741439b0 2024-03-29T11:12:25.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.650+0000 7fcc70ff9640 10 monclient: _finish_auth 0 2024-03-29T11:12:25.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.650+0000 7fcc70ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:12:25.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.650+0000 7fcc70ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:25.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.651+0000 7fcc7a7fb640 5 monclient: authenticate success, global_id 4342 2024-03-29T11:12:25.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.651+0000 7fcc7a7fb640 10 monclient: _renew_subs 2024-03-29T11:12:25.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.651+0000 7fcc7a7fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:12:25.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.651+0000 7fcc7a7fb640 1 -- 172.21.15.18:0/1101905440 --> v1:172.21.15.70:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc74120430 con 0x7fcc741439b0 2024-03-29T11:12:25.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.651+0000 7fcc7a7fb640 10 client.4342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:25.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.651+0000 7fcc7a7fb640 10 monclient: _renew_subs 2024-03-29T11:12:25.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.651+0000 7fcc7a7fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:12:25.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.651+0000 7fcc7a7fb640 1 -- 172.21.15.18:0/1101905440 --> v1:172.21.15.70:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc74120160 con 0x7fcc741439b0 2024-03-29T11:12:25.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.651+0000 7fcc7a7fb640 1 librados: init done 2024-03-29T11:12:25.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.652+0000 7fcc7a7fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:25.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.652+0000 7fcc7a7fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:25.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.652+0000 7fcc7a7fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:12:25.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.652+0000 7fcc7a7fb640 1 -- 172.21.15.18:0/1101905440 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc74075f10 con 0x7fcc741439b0 2024-03-29T11:12:25.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.654+0000 7fcc70ff9640 1 -- 172.21.15.18:0/1101905440 <== mon.1 v1:172.21.15.70:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc6c001a00 con 0x7fcc741439b0 2024-03-29T11:12:25.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.654+0000 7fcc70ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:25.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.654+0000 7fcc70ff9640 1 -- 172.21.15.18:0/1101905440 <== mon.1 v1:172.21.15.70:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fcc6c003b50 con 0x7fcc741439b0 2024-03-29T11:12:25.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.654+0000 7fcc727fc640 4 set_mon_vals no callback set 2024-03-29T11:12:25.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.654+0000 7fcc70ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:25.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.654+0000 7fcc70ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:25.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.654+0000 7fcc70ff9640 10 monclient: dump: 2024-03-29T11:12:25.640 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:12:25.640 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:25.640 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:25.640 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:25.641 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:25.641 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:25.641 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:12:25.641 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:12:25.641 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:12:25.641 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:25.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.657+0000 7fcc70ff9640 1 -- 172.21.15.18:0/1101905440 <== mon.1 v1:172.21.15.70:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7fcc6c019ba0 con 0x7fcc741439b0 2024-03-29T11:12:25.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.657+0000 7fcc70ff9640 1 -- 172.21.15.18:0/1101905440 <== mon.1 v1:172.21.15.70:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7fcc6c0838a0 con 0x7fcc741439b0 2024-03-29T11:12:25.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.657+0000 7fcc70ff9640 10 client.4342.objecter ms_dispatch 0x7fcc74000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:12:25.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.657+0000 7fcc70ff9640 3 client.4342.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:12:25.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.657+0000 7fcc70ff9640 3 client.4342.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:12:25.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.657+0000 7fcc70ff9640 20 client.4342.objecter dump_active .. 0 homeless 2024-03-29T11:12:25.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.657+0000 7fcc73fff640 10 monclient: get_auth_request con 0x7fcc6006e420 auth_method 0 2024-03-29T11:12:25.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.658+0000 7fcc70ff9640 10 client.4342.objecter ms_handle_connect 0x7fcc6006e420 2024-03-29T11:12:25.643 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.661+0000 7fcc70ff9640 1 -- 172.21.15.18:0/1101905440 <== mon.1 v1:172.21.15.70:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcc6c04efb0 con 0x7fcc741439b0 2024-03-29T11:12:25.643 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.661+0000 7fcc70ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:25.643 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:25.661+0000 7fcc70ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:25.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:26.012+0000 7fcc7a7fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:25.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:26.012+0000 7fcc7a7fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:25.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:26.012+0000 7fcc7a7fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:12:25.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:26.012+0000 7fcc7a7fb640 1 -- 172.21.15.18:0/1101905440 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc74146540 con 0x7fcc741439b0 2024-03-29T11:12:25.992 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:12:25.993 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:25.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:26.013+0000 7fcc70ff9640 1 -- 172.21.15.18:0/1101905440 <== mon.1 v1:172.21.15.70:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7fcc74146540 con 0x7fcc741439b0 2024-03-29T11:12:25.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:26.013+0000 7fcc70ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:25.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:26.013+0000 7fcc70ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:25.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:26.016+0000 7fcc7a7fb640 10 librados: watch_flush enter 2024-03-29T11:12:25.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:26.016+0000 7fcc7a7fb640 10 librados: watch_flush exit 2024-03-29T11:12:25.999 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:26.016+0000 7fcc7a7fb640 20 client.4342.objecter shutdown clearing up homeless session... 2024-03-29T11:12:25.999 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:26.016+0000 7fcc7a7fb640 10 client.4342.objecter successfully canceled tick 2024-03-29T11:12:25.999 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:26.016+0000 7fcc7a7fb640 1 -- 172.21.15.18:0/1101905440 >> v1:172.21.15.70:6816/3101871562 conn(0x7fcc6006e420 legacy=0x7fcc600708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:25.999 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:26.016+0000 7fcc7a7fb640 10 monclient: shutdown 2024-03-29T11:12:25.999 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:26.016+0000 7fcc7a7fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:25.999 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:26.017+0000 7fcc7a7fb640 1 -- 172.21.15.18:0/1101905440 >> v1:172.21.15.70:6789/0 conn(0x7fcc741439b0 legacy=0x7fcc741402d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:25.999 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:26.017+0000 7fcc7a7fb640 1 -- 172.21.15.18:0/1101905440 shutdown_connections 2024-03-29T11:12:25.999 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:26.017+0000 7fcc7a7fb640 1 -- 172.21.15.18:0/1101905440 >> 172.21.15.18:0/1101905440 conn(0x7fcc74066a00 msgr2=0x7fcc74066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:25.999 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:26.017+0000 7fcc7a7fb640 1 -- 172.21.15.18:0/1101905440 shutdown_connections 2024-03-29T11:12:25.999 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:26.017+0000 7fcc7a7fb640 1 -- 172.21.15.18:0/1101905440 wait complete. 2024-03-29T11:12:25.999 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:26.017+0000 7fcc7a7fb640 1 librados: shutdown 2024-03-29T11:12:26.011 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:12:32.820 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:32.966 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.988+0000 7f536157b640 10 monclient: get_monmap_and_config 2024-03-29T11:12:32.966 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.988+0000 7f536157b640 10 monclient: build_initial_monmap 2024-03-29T11:12:32.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.988+0000 7f536157b640 10 monclient: monmap: 2024-03-29T11:12:32.967 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:12:32.967 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:32.967 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:12:32.989335+0000 2024-03-29T11:12:32.967 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:12:32.989335+0000 2024-03-29T11:12:32.967 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:32.967 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:32.967 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:12:32.967 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:12:32.967 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:12:32.967 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:32.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.989+0000 7f536157b640 1 Processor -- start 2024-03-29T11:12:32.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.989+0000 7f536157b640 1 -- start start 2024-03-29T11:12:32.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.989+0000 7f536157b640 10 monclient: init 2024-03-29T11:12:32.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.990+0000 7f536157b640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:32.969 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.991+0000 7f536157b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:12:32.969 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.991+0000 7f536157b640 10 monclient(hunting): picked mon.noname-c con 0x7f535c0b0860 addr v1:172.21.15.70:6789/0 2024-03-29T11:12:32.970 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.991+0000 7f536157b640 10 monclient(hunting): picked mon.noname-a con 0x7f535c13ef30 addr v1:172.21.15.18:6789/0 2024-03-29T11:12:32.970 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.991+0000 7f536157b640 10 monclient(hunting): picked mon.noname-b con 0x7f535c0af8e0 addr v1:172.21.15.18:6790/0 2024-03-29T11:12:32.970 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.991+0000 7f53527fc640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f535c13ef30 0x7f535c0af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:39432/0 (socket says 172.21.15.18:39432) 2024-03-29T11:12:32.970 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.991+0000 7f53527fc640 1 -- 172.21.15.18:0/3729428936 learned_addr learned my addr 172.21.15.18:0/3729428936 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:12:32.970 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.991+0000 7f535affd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f535c0b0860 0x7f535c13e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.18:44748/0 (socket says 172.21.15.18:44748) 2024-03-29T11:12:32.970 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.991+0000 7f536157b640 1 -- 172.21.15.18:0/3729428936 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f535c0b0c40 con 0x7f535c13ef30 2024-03-29T11:12:32.970 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.991+0000 7f536157b640 1 -- 172.21.15.18:0/3729428936 --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f535c0aed10 con 0x7f535c0b0860 2024-03-29T11:12:32.970 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.991+0000 7f536157b640 1 -- 172.21.15.18:0/3729428936 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f535c0afd50 con 0x7f535c0af8e0 2024-03-29T11:12:32.970 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.991+0000 7f536157b640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:32.970 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.991+0000 7f536157b640 10 monclient(hunting): authenticate will time out at 979.263693s 2024-03-29T11:12:32.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.992+0000 7f535a7fc640 1 -- 172.21.15.18:0/3729428936 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 644836133 0 0) 0x7f535c0afd50 con 0x7f535c0af8e0 2024-03-29T11:12:32.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.992+0000 7f535a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:12:32.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.992+0000 7f535a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:32.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.992+0000 7f535a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:32.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.992+0000 7f535a7fc640 10 monclient(hunting): my global_id is 4373 2024-03-29T11:12:32.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.992+0000 7f535a7fc640 1 -- 172.21.15.18:0/3729428936 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5348003520 con 0x7f535c0af8e0 2024-03-29T11:12:32.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.992+0000 7f535a7fc640 1 -- 172.21.15.18:0/3729428936 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 289521957 0 0) 0x7f535c0b0c40 con 0x7f535c13ef30 2024-03-29T11:12:32.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.992+0000 7f535a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:12:32.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.992+0000 7f535a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:32.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.992+0000 7f535a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:32.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.992+0000 7f535a7fc640 10 monclient(hunting): my global_id is 4341 2024-03-29T11:12:32.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.992+0000 7f535a7fc640 1 -- 172.21.15.18:0/3729428936 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f535c0afd50 con 0x7f535c13ef30 2024-03-29T11:12:32.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.992+0000 7f535a7fc640 1 -- 172.21.15.18:0/3729428936 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877481052 0 0) 0x7f535c0aed10 con 0x7f535c0b0860 2024-03-29T11:12:32.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.992+0000 7f535a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:12:32.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.992+0000 7f535a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:32.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.992+0000 7f535a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:32.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.992+0000 7f535a7fc640 10 monclient(hunting): my global_id is 4345 2024-03-29T11:12:32.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.992+0000 7f535a7fc640 1 -- 172.21.15.18:0/3729428936 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f535c0b0c40 con 0x7f535c0b0860 2024-03-29T11:12:32.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.993+0000 7f535a7fc640 1 -- 172.21.15.18:0/3729428936 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3075969486 0 0) 0x7f535c0b0c40 con 0x7f535c0b0860 2024-03-29T11:12:32.973 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.993+0000 7f535a7fc640 1 -- 172.21.15.18:0/3729428936 >> v1:172.21.15.18:6790/0 conn(0x7f535c0af8e0 legacy=0x7f535c149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:32.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.993+0000 7f535a7fc640 1 -- 172.21.15.18:0/3729428936 >> v1:172.21.15.18:6789/0 conn(0x7f535c13ef30 legacy=0x7f535c0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:32.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.993+0000 7f535a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:12:32.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.993+0000 7f535a7fc640 1 monclient: found mon.noname-c 2024-03-29T11:12:32.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.993+0000 7f535a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:32.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.993+0000 7f535a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.70:6789/0 2024-03-29T11:12:32.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.993+0000 7f535a7fc640 1 -- 172.21.15.18:0/3729428936 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f535c0ab1b0 con 0x7f535c0b0860 2024-03-29T11:12:32.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.993+0000 7f535a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:12:32.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.993+0000 7f535a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:12:32.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.993+0000 7f535a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:32.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.993+0000 7f535a7fc640 1 -- 172.21.15.18:0/3729428936 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f534c0015b0 con 0x7f535c0b0860 2024-03-29T11:12:32.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.993+0000 7f535a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:32.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.993+0000 7f535a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:12:32.974 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.993+0000 7f535a7fc640 10 monclient: dump: 2024-03-29T11:12:32.974 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:12:32.975 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:32.975 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:32.975 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:32.975 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:32.975 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:32.975 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:12:32.975 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:12:32.975 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:12:32.975 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:32.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.993+0000 7f536157b640 5 monclient: authenticate success, global_id 4345 2024-03-29T11:12:32.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.993+0000 7f536157b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:32.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.994+0000 7f535a7fc640 1 -- 172.21.15.18:0/3729428936 <== mon.1 v1:172.21.15.70:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f534c001710 con 0x7f535c0b0860 2024-03-29T11:12:32.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.994+0000 7f535a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:32.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.994+0000 7f535a7fc640 1 -- 172.21.15.18:0/3729428936 <== mon.1 v1:172.21.15.70:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f534c001b60 con 0x7f535c0b0860 2024-03-29T11:12:32.975 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.994+0000 7f535a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:32.976 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.994+0000 7f535a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:32.976 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.994+0000 7f535a7fc640 10 monclient: dump: 2024-03-29T11:12:32.976 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:12:32.976 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:32.976 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:32.976 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:32.976 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:32.976 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:32.976 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:12:32.976 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:12:32.976 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:12:32.976 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:32.976 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.994+0000 7f536157b640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:32.977 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.994+0000 7f536157b640 4 set_mon_vals no callback set 2024-03-29T11:12:32.983 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.995+0000 7f536157b640 10 monclient: shutdown 2024-03-29T11:12:32.983 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.995+0000 7f536157b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:32.983 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.995+0000 7f536157b640 1 -- 172.21.15.18:0/3729428936 >> v1:172.21.15.70:6789/0 conn(0x7f535c0b0860 legacy=0x7f535c13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:32.983 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.995+0000 7f536157b640 1 -- 172.21.15.18:0/3729428936 shutdown_connections 2024-03-29T11:12:32.983 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.995+0000 7f536157b640 1 -- 172.21.15.18:0/3729428936 >> 172.21.15.18:0/3729428936 conn(0x7f535c06b660 msgr2=0x7f535c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:32.983 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.995+0000 7f536157b640 1 -- 172.21.15.18:0/3729428936 shutdown_connections 2024-03-29T11:12:32.983 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.995+0000 7f536157b640 1 -- 172.21.15.18:0/3729428936 wait complete. 2024-03-29T11:12:32.983 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.996+0000 7f536157b640 10 monclient: build_initial_monmap 2024-03-29T11:12:32.983 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.997+0000 7f536157b640 10 monclient: monmap: 2024-03-29T11:12:32.984 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:12:32.984 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:32.984 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:12:32.984 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:12:32.984 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:32.984 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:32.984 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:12:32.984 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:12:32.984 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:12:32.984 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:32.984 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.997+0000 7f536157b640 1 librados: starting msgr at 2024-03-29T11:12:32.984 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.997+0000 7f536157b640 1 librados: starting objecter 2024-03-29T11:12:32.984 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.997+0000 7f536157b640 1 Processor -- start 2024-03-29T11:12:32.984 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.998+0000 7f536157b640 1 -- start start 2024-03-29T11:12:32.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.998+0000 7f536157b640 1 librados: setting wanted keys 2024-03-29T11:12:32.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.998+0000 7f536157b640 1 librados: calling monclient init 2024-03-29T11:12:32.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.998+0000 7f536157b640 10 monclient: init 2024-03-29T11:12:32.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.999+0000 7f536157b640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:32.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.999+0000 7f536157b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:12:32.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.999+0000 7f536157b640 10 monclient(hunting): picked mon.noname-a con 0x7f535c0af8e0 addr v1:172.21.15.18:6789/0 2024-03-29T11:12:32.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.999+0000 7f536157b640 10 monclient(hunting): picked mon.noname-c con 0x7f535c0b0860 addr v1:172.21.15.18:6790/0 2024-03-29T11:12:32.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.999+0000 7f536157b640 10 monclient(hunting): picked mon.noname-b con 0x7f535c13ef30 addr v1:172.21.15.70:6789/0 2024-03-29T11:12:32.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.999+0000 7f536157b640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f535c0b3ac0 con 0x7f535c0af8e0 2024-03-29T11:12:32.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.999+0000 7f536157b640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f535c0b3c50 con 0x7f535c13ef30 2024-03-29T11:12:32.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.999+0000 7f536157b640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f535c0b3de0 con 0x7f535c0b0860 2024-03-29T11:12:32.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.999+0000 7f536157b640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:32.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.999+0000 7f536157b640 10 monclient(hunting): authenticate will time out at 979.271639s 2024-03-29T11:12:32.985 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.999+0000 7f53527fc640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f535c0b0860 0x7f535c1d6430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:33428/0 (socket says 172.21.15.18:33428) 2024-03-29T11:12:32.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.999+0000 7f53527fc640 1 -- 172.21.15.18:0/3795330887 learned_addr learned my addr 172.21.15.18:0/3795330887 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:12:32.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.999+0000 7f5353fff640 10 client.?.objecter ms_handle_connect 0x7f535c0b0860 2024-03-29T11:12:32.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:32.999+0000 7f5353fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:32.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.000+0000 7f5353fff640 1 -- 172.21.15.18:0/3795330887 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3676372288 0 0) 0x7f5344001130 con 0x7f535c0b0860 2024-03-29T11:12:32.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.000+0000 7f5353fff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:32.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.000+0000 7f5353fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:32.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.000+0000 7f5353fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:32.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.000+0000 7f5353fff640 10 monclient(hunting): my global_id is 4376 2024-03-29T11:12:32.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.000+0000 7f5353fff640 1 -- 172.21.15.18:0/3795330887 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f533c004eb0 con 0x7f535c0b0860 2024-03-29T11:12:32.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.000+0000 7f5353fff640 10 client.?.objecter ms_handle_connect 0x7f535c0af8e0 2024-03-29T11:12:32.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.000+0000 7f5353fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:32.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.000+0000 7f5353fff640 1 -- 172.21.15.18:0/3795330887 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2130707175 0 0) 0x7f535c0b3ac0 con 0x7f535c0af8e0 2024-03-29T11:12:32.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.000+0000 7f5353fff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:32.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.000+0000 7f5353fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:32.986 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.000+0000 7f5353fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:32.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.001+0000 7f5353fff640 10 monclient(hunting): my global_id is 4344 2024-03-29T11:12:32.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.001+0000 7f5353fff640 1 -- 172.21.15.18:0/3795330887 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5344001130 con 0x7f535c0af8e0 2024-03-29T11:12:32.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.001+0000 7f5353fff640 10 client.?.objecter ms_handle_connect 0x7f535c13ef30 2024-03-29T11:12:32.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.001+0000 7f5353fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:32.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.001+0000 7f5353fff640 1 -- 172.21.15.18:0/3795330887 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1977052190 0 0) 0x7f533c004eb0 con 0x7f535c0b0860 2024-03-29T11:12:32.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.001+0000 7f5353fff640 1 -- 172.21.15.18:0/3795330887 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f535c0b3ac0 con 0x7f535c0b0860 2024-03-29T11:12:32.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.001+0000 7f5353fff640 1 -- 172.21.15.18:0/3795330887 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f53440018a0 con 0x7f535c0b0860 2024-03-29T11:12:32.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.001+0000 7f5353fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:32.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.001+0000 7f5353fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:12:32.987 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.001+0000 7f5353fff640 10 monclient(hunting): dump: 2024-03-29T11:12:32.987 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:12:32.987 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:32.987 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:32.987 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:32.987 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:32.988 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:32.988 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:12:32.988 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:12:32.988 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:12:32.988 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:32.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.001+0000 7f5353fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:32.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.001+0000 7f5353fff640 1 -- 172.21.15.18:0/3795330887 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1586167524 0 0) 0x7f535c0b3c50 con 0x7f535c13ef30 2024-03-29T11:12:32.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.001+0000 7f5353fff640 10 monclient(hunting): _negotiate 2024-03-29T11:12:32.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.001+0000 7f5353fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:32.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.001+0000 7f5353fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:32.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.001+0000 7f5353fff640 10 monclient(hunting): my global_id is 4348 2024-03-29T11:12:32.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.001+0000 7f5353fff640 1 -- 172.21.15.18:0/3795330887 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f533c004eb0 con 0x7f535c13ef30 2024-03-29T11:12:32.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.001+0000 7f5353fff640 1 -- 172.21.15.18:0/3795330887 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1942983935 0 0) 0x7f535c0b3ac0 con 0x7f535c0b0860 2024-03-29T11:12:32.988 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.001+0000 7f5353fff640 1 -- 172.21.15.18:0/3795330887 >> v1:172.21.15.70:6789/0 conn(0x7f535c13ef30 legacy=0x7f535c1368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:32.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.001+0000 7f5353fff640 1 -- 172.21.15.18:0/3795330887 >> v1:172.21.15.18:6789/0 conn(0x7f535c0af8e0 legacy=0x7f535c0b2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:32.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.001+0000 7f5353fff640 10 monclient: _finish_hunting 0 2024-03-29T11:12:32.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.002+0000 7f5353fff640 1 monclient: found mon.c 2024-03-29T11:12:32.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.002+0000 7f5353fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:32.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.002+0000 7f5353fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:12:32.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.002+0000 7f5353fff640 1 -- 172.21.15.18:0/3795330887 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f535c0b34e0 con 0x7f535c0b0860 2024-03-29T11:12:32.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.002+0000 7f5353fff640 10 monclient: _finish_auth 0 2024-03-29T11:12:32.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.002+0000 7f5353fff640 10 monclient: _check_auth_tickets 2024-03-29T11:12:32.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.002+0000 7f5353fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:32.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.002+0000 7f536157b640 5 monclient: authenticate success, global_id 4376 2024-03-29T11:12:32.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.002+0000 7f536157b640 10 monclient: _renew_subs 2024-03-29T11:12:32.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.002+0000 7f536157b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:12:32.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.002+0000 7f536157b640 1 -- 172.21.15.18:0/3795330887 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f535c0b3730 con 0x7f535c0b0860 2024-03-29T11:12:32.989 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.002+0000 7f536157b640 10 client.4376.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:32.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.002+0000 7f536157b640 10 monclient: _renew_subs 2024-03-29T11:12:32.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.002+0000 7f536157b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:12:32.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.002+0000 7f536157b640 1 -- 172.21.15.18:0/3795330887 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f535c1d6d20 con 0x7f535c0b0860 2024-03-29T11:12:32.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.002+0000 7f536157b640 1 librados: init done 2024-03-29T11:12:32.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.003+0000 7f5353fff640 1 -- 172.21.15.18:0/3795330887 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53440047e0 con 0x7f535c0b0860 2024-03-29T11:12:32.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.003+0000 7f5353fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:32.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.003+0000 7f5359ffb640 4 set_mon_vals no callback set 2024-03-29T11:12:32.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.003+0000 7f5353fff640 1 -- 172.21.15.18:0/3795330887 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f5344004af0 con 0x7f535c0b0860 2024-03-29T11:12:32.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.003+0000 7f5353fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:32.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.003+0000 7f5353fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:32.990 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.003+0000 7f5353fff640 10 monclient: dump: 2024-03-29T11:12:32.990 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:12:32.990 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:32.991 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:32.991 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:32.991 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:32.991 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:32.991 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:12:32.991 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:12:32.991 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:12:32.991 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:32.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.004+0000 7f536157b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:32.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.004+0000 7f536157b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:32.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.004+0000 7f536157b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:12:32.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.004+0000 7f536157b640 1 -- 172.21.15.18:0/3795330887 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5340005080 con 0x7f535c0b0860 2024-03-29T11:12:32.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.006+0000 7f5353fff640 1 -- 172.21.15.18:0/3795330887 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f534401aac0 con 0x7f535c0b0860 2024-03-29T11:12:32.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.006+0000 7f5353fff640 1 -- 172.21.15.18:0/3795330887 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f5344084980 con 0x7f535c0b0860 2024-03-29T11:12:32.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.006+0000 7f535affd640 10 monclient: get_auth_request con 0x7f533c06dc90 auth_method 0 2024-03-29T11:12:32.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.006+0000 7f5353fff640 10 client.4376.objecter ms_dispatch 0x7f535c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:12:32.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.006+0000 7f5353fff640 3 client.4376.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:12:32.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.006+0000 7f5353fff640 3 client.4376.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:12:32.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.007+0000 7f5353fff640 20 client.4376.objecter dump_active .. 0 homeless 2024-03-29T11:12:32.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.007+0000 7f5353fff640 10 client.4376.objecter ms_handle_connect 0x7f533c06dc90 2024-03-29T11:12:32.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.015+0000 7f5353fff640 1 -- 172.21.15.18:0/3795330887 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5344050260 con 0x7f535c0b0860 2024-03-29T11:12:32.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.015+0000 7f5353fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:32.994 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.015+0000 7f5353fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:33.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.420+0000 7f536157b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:33.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.420+0000 7f536157b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:33.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.420+0000 7f536157b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:12:33.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.420+0000 7f536157b640 1 -- 172.21.15.18:0/3795330887 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5340005330 con 0x7f535c0b0860 2024-03-29T11:12:33.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.421+0000 7f5353fff640 1 -- 172.21.15.18:0/3795330887 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f5344052190 con 0x7f535c0b0860 2024-03-29T11:12:33.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.421+0000 7f5353fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:33.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.421+0000 7f5353fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:33.399 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:12:33.400 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:33.403 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.425+0000 7f536157b640 10 librados: watch_flush enter 2024-03-29T11:12:33.403 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.425+0000 7f536157b640 10 librados: watch_flush exit 2024-03-29T11:12:33.403 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.425+0000 7f536157b640 20 client.4376.objecter shutdown clearing up homeless session... 2024-03-29T11:12:33.403 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.425+0000 7f536157b640 10 client.4376.objecter successfully canceled tick 2024-03-29T11:12:33.403 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.425+0000 7f536157b640 1 -- 172.21.15.18:0/3795330887 >> v1:172.21.15.70:6816/3101871562 conn(0x7f533c06dc90 legacy=0x7f533c070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:33.403 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.425+0000 7f536157b640 10 monclient: shutdown 2024-03-29T11:12:33.403 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.425+0000 7f536157b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:33.403 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.425+0000 7f536157b640 1 -- 172.21.15.18:0/3795330887 >> v1:172.21.15.18:6790/0 conn(0x7f535c0b0860 legacy=0x7f535c1d6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:33.403 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.425+0000 7f536157b640 1 -- 172.21.15.18:0/3795330887 shutdown_connections 2024-03-29T11:12:33.403 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.425+0000 7f536157b640 1 -- 172.21.15.18:0/3795330887 >> 172.21.15.18:0/3795330887 conn(0x7f535c06b660 msgr2=0x7f535c13e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:33.404 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.426+0000 7f536157b640 1 -- 172.21.15.18:0/3795330887 shutdown_connections 2024-03-29T11:12:33.404 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.426+0000 7f536157b640 1 -- 172.21.15.18:0/3795330887 wait complete. 2024-03-29T11:12:33.404 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:33.426+0000 7f536157b640 1 librados: shutdown 2024-03-29T11:12:33.418 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:12:40.225 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:40.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.396+0000 7f21a4177640 10 monclient: get_monmap_and_config 2024-03-29T11:12:40.375 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.396+0000 7f21a4177640 10 monclient: build_initial_monmap 2024-03-29T11:12:40.375 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.396+0000 7f21a4177640 10 monclient: monmap: 2024-03-29T11:12:40.375 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:12:40.375 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:40.375 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:12:40.397772+0000 2024-03-29T11:12:40.375 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:12:40.397772+0000 2024-03-29T11:12:40.375 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:40.375 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:40.375 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:12:40.375 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:12:40.375 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:12:40.375 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:40.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.398+0000 7f21a4177640 1 Processor -- start 2024-03-29T11:12:40.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.398+0000 7f21a4177640 1 -- start start 2024-03-29T11:12:40.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.398+0000 7f21a4177640 10 monclient: init 2024-03-29T11:12:40.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.400+0000 7f21a4177640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:40.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.400+0000 7f21a4177640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:12:40.384 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.400+0000 7f21a4177640 10 monclient(hunting): picked mon.noname-a con 0x7f219c0a9750 addr v1:172.21.15.18:6789/0 2024-03-29T11:12:40.384 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.400+0000 7f21a4177640 10 monclient(hunting): picked mon.noname-b con 0x7f219c13e1e0 addr v1:172.21.15.18:6790/0 2024-03-29T11:12:40.384 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.400+0000 7f21a4177640 10 monclient(hunting): picked mon.noname-c con 0x7f219c071c30 addr v1:172.21.15.70:6789/0 2024-03-29T11:12:40.384 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.400+0000 7f21a4177640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f219c0ab300 con 0x7f219c0a9750 2024-03-29T11:12:40.384 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.400+0000 7f21a4177640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f219c0ac2f0 con 0x7f219c071c30 2024-03-29T11:12:40.384 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.400+0000 7f21a4177640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f219c0ad8d0 con 0x7f219c13e1e0 2024-03-29T11:12:40.384 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.400+0000 7f21a4177640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:40.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.400+0000 7f21a4177640 10 monclient(hunting): authenticate will time out at 986.672738s 2024-03-29T11:12:40.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.402+0000 7f21a1712640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f219c13e1e0 0x7f219c0ac710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:47430/0 (socket says 172.21.15.18:47430) 2024-03-29T11:12:40.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.402+0000 7f21a1712640 1 -- 172.21.15.18:0/831619825 learned_addr learned my addr 172.21.15.18:0/831619825 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:12:40.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.402+0000 7f21a0f11640 1 -- 172.21.15.18:0/831619825 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2445685784 0 0) 0x7f219c0ad8d0 con 0x7f219c13e1e0 2024-03-29T11:12:40.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.402+0000 7f21a0f11640 10 monclient(hunting): _negotiate 2024-03-29T11:12:40.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.402+0000 7f21a0f11640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:40.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.402+0000 7f21a0f11640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:40.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.402+0000 7f21a0f11640 10 monclient(hunting): my global_id is 4379 2024-03-29T11:12:40.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.402+0000 7f21a0f11640 1 -- 172.21.15.18:0/831619825 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2188003520 con 0x7f219c13e1e0 2024-03-29T11:12:40.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.403+0000 7f21a0f11640 1 -- 172.21.15.18:0/831619825 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1171022510 0 0) 0x7f2188003520 con 0x7f219c13e1e0 2024-03-29T11:12:40.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.403+0000 7f21a0f11640 1 -- 172.21.15.18:0/831619825 >> v1:172.21.15.70:6789/0 conn(0x7f219c071c30 legacy=0x7f219c0ace20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:12:40.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.404+0000 7f21a0f11640 1 -- 172.21.15.18:0/831619825 >> v1:172.21.15.18:6789/0 conn(0x7f219c0a9750 legacy=0x7f219c13dad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:12:40.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.405+0000 7f21a0f11640 10 monclient: _finish_hunting 0 2024-03-29T11:12:40.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.405+0000 7f21a0f11640 1 monclient: found mon.noname-b 2024-03-29T11:12:40.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.405+0000 7f21a0f11640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:40.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.405+0000 7f21a0f11640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:12:40.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.405+0000 7f21a0f11640 1 -- 172.21.15.18:0/831619825 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f219c0ada60 con 0x7f219c13e1e0 2024-03-29T11:12:40.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.405+0000 7f21a0f11640 10 monclient: _finish_auth 0 2024-03-29T11:12:40.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.405+0000 7f21a0f11640 10 monclient: _check_auth_tickets 2024-03-29T11:12:40.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.405+0000 7f21a0f11640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:40.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.405+0000 7f21a0f11640 1 -- 172.21.15.18:0/831619825 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f218c001690 con 0x7f219c13e1e0 2024-03-29T11:12:40.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.405+0000 7f21a0f11640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:40.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.405+0000 7f21a0f11640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:40.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.405+0000 7f21a0f11640 10 monclient: dump: 2024-03-29T11:12:40.388 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:12:40.388 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:40.388 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:40.388 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:40.389 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:40.389 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:40.389 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:12:40.389 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:12:40.389 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:12:40.389 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:40.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.406+0000 7f21a0f11640 1 -- 172.21.15.18:0/831619825 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f218c001860 con 0x7f219c13e1e0 2024-03-29T11:12:40.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.406+0000 7f21a0f11640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:40.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.406+0000 7f21a0f11640 1 -- 172.21.15.18:0/831619825 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f218c001cb0 con 0x7f219c13e1e0 2024-03-29T11:12:40.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.406+0000 7f21a0f11640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:40.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.406+0000 7f21a0f11640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:40.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.406+0000 7f21a0f11640 10 monclient: dump: 2024-03-29T11:12:40.389 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:12:40.389 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:40.390 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:40.390 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:40.390 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:40.390 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:40.390 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:12:40.390 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:12:40.390 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:12:40.390 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:40.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.406+0000 7f21a4177640 5 monclient: authenticate success, global_id 4379 2024-03-29T11:12:40.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.406+0000 7f21a4177640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:40.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.406+0000 7f21a4177640 4 set_mon_vals no callback set 2024-03-29T11:12:40.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.407+0000 7f21a4177640 10 monclient: shutdown 2024-03-29T11:12:40.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.407+0000 7f21a4177640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:40.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.407+0000 7f21a4177640 1 -- 172.21.15.18:0/831619825 >> v1:172.21.15.18:6790/0 conn(0x7f219c13e1e0 legacy=0x7f219c0ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:40.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.408+0000 7f21a4177640 1 -- 172.21.15.18:0/831619825 shutdown_connections 2024-03-29T11:12:40.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.408+0000 7f21a4177640 1 -- 172.21.15.18:0/831619825 >> 172.21.15.18:0/831619825 conn(0x7f219c06b660 msgr2=0x7f219c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:40.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.409+0000 7f21a4177640 1 -- 172.21.15.18:0/831619825 shutdown_connections 2024-03-29T11:12:40.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.409+0000 7f21a4177640 1 -- 172.21.15.18:0/831619825 wait complete. 2024-03-29T11:12:40.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.410+0000 7f21a4177640 10 monclient: build_initial_monmap 2024-03-29T11:12:40.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.410+0000 7f21a4177640 10 monclient: monmap: 2024-03-29T11:12:40.391 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:12:40.391 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:40.391 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:12:40.391 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:12:40.391 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:40.391 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:40.391 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:12:40.391 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:12:40.391 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:12:40.391 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:40.392 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.410+0000 7f21a4177640 1 librados: starting msgr at 2024-03-29T11:12:40.392 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.410+0000 7f21a4177640 1 librados: starting objecter 2024-03-29T11:12:40.392 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.411+0000 7f21a4177640 1 Processor -- start 2024-03-29T11:12:40.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.411+0000 7f21a4177640 1 -- start start 2024-03-29T11:12:40.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.411+0000 7f21a4177640 1 librados: setting wanted keys 2024-03-29T11:12:40.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.411+0000 7f21a4177640 1 librados: calling monclient init 2024-03-29T11:12:40.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.411+0000 7f21a4177640 10 monclient: init 2024-03-29T11:12:40.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.412+0000 7f21a4177640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:40.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.412+0000 7f21a4177640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:12:40.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.412+0000 7f21a4177640 10 monclient(hunting): picked mon.noname-b con 0x7f219c071c30 addr v1:172.21.15.70:6789/0 2024-03-29T11:12:40.393 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.412+0000 7f21a4177640 10 monclient(hunting): picked mon.noname-a con 0x7f219c0a9750 addr v1:172.21.15.18:6789/0 2024-03-29T11:12:40.394 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.412+0000 7f21a4177640 10 monclient(hunting): picked mon.noname-c con 0x7f219c13e1e0 addr v1:172.21.15.18:6790/0 2024-03-29T11:12:40.394 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.412+0000 7f21a4177640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f219c136e10 con 0x7f219c0a9750 2024-03-29T11:12:40.394 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.412+0000 7f21a4177640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f219c140ef0 con 0x7f219c071c30 2024-03-29T11:12:40.394 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.412+0000 7f21a4177640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f219c141c70 con 0x7f219c13e1e0 2024-03-29T11:12:40.394 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.412+0000 7f21a4177640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:40.394 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.412+0000 7f21a4177640 10 monclient(hunting): authenticate will time out at 986.684513s 2024-03-29T11:12:40.394 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.412+0000 7f21a2714640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f219c13e1e0 0x7f219c1ee920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:47956/0 (socket says 172.21.15.18:47956) 2024-03-29T11:12:40.394 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.412+0000 7f21a2714640 1 -- 172.21.15.18:0/2650913168 learned_addr learned my addr 172.21.15.18:0/2650913168 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:12:40.394 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.413+0000 7f21927fc640 10 client.?.objecter ms_handle_connect 0x7f219c13e1e0 2024-03-29T11:12:40.394 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.413+0000 7f21927fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:40.394 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.413+0000 7f21927fc640 1 -- 172.21.15.18:0/2650913168 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2344808345 0 0) 0x7f219c141c70 con 0x7f219c13e1e0 2024-03-29T11:12:40.394 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.413+0000 7f21927fc640 10 monclient(hunting): _negotiate 2024-03-29T11:12:40.394 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.413+0000 7f21927fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:40.394 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.413+0000 7f21927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:40.395 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.413+0000 7f21927fc640 10 monclient(hunting): my global_id is 4382 2024-03-29T11:12:40.395 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.413+0000 7f21927fc640 1 -- 172.21.15.18:0/2650913168 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2180004eb0 con 0x7f219c13e1e0 2024-03-29T11:12:40.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.416+0000 7f21927fc640 1 -- 172.21.15.18:0/2650913168 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 993778030 0 0) 0x7f2180004eb0 con 0x7f219c13e1e0 2024-03-29T11:12:40.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.416+0000 7f21927fc640 1 -- 172.21.15.18:0/2650913168 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f219c141c70 con 0x7f219c13e1e0 2024-03-29T11:12:40.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.416+0000 7f21927fc640 1 -- 172.21.15.18:0/2650913168 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f2194001c60 con 0x7f219c13e1e0 2024-03-29T11:12:40.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.416+0000 7f21927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:40.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.416+0000 7f21927fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:12:40.397 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.416+0000 7f21927fc640 10 monclient(hunting): dump: 2024-03-29T11:12:40.397 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:12:40.397 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:40.397 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:40.397 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:40.397 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:40.397 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:40.398 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:12:40.398 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:12:40.398 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:12:40.398 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:40.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.416+0000 7f21927fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:40.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.417+0000 7f21927fc640 10 client.?.objecter ms_handle_connect 0x7f219c0a9750 2024-03-29T11:12:40.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.417+0000 7f21927fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:40.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.417+0000 7f21927fc640 1 -- 172.21.15.18:0/2650913168 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1952531065 0 0) 0x7f219c136e10 con 0x7f219c0a9750 2024-03-29T11:12:40.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.417+0000 7f21927fc640 10 monclient(hunting): _negotiate 2024-03-29T11:12:40.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.417+0000 7f21927fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:40.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.417+0000 7f21927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:40.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.417+0000 7f21927fc640 10 monclient(hunting): my global_id is 4347 2024-03-29T11:12:40.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.417+0000 7f21927fc640 1 -- 172.21.15.18:0/2650913168 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2180004eb0 con 0x7f219c0a9750 2024-03-29T11:12:40.398 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.417+0000 7f21927fc640 1 -- 172.21.15.18:0/2650913168 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4138666563 0 0) 0x7f219c141c70 con 0x7f219c13e1e0 2024-03-29T11:12:40.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.418+0000 7f21927fc640 1 -- 172.21.15.18:0/2650913168 >> v1:172.21.15.70:6789/0 conn(0x7f219c071c30 legacy=0x7f219c1e7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:40.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.418+0000 7f21927fc640 1 -- 172.21.15.18:0/2650913168 >> v1:172.21.15.18:6789/0 conn(0x7f219c0a9750 legacy=0x7f219c1e7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:40.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.418+0000 7f21927fc640 10 monclient: _finish_hunting 0 2024-03-29T11:12:40.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.418+0000 7f21927fc640 1 monclient: found mon.c 2024-03-29T11:12:40.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.418+0000 7f21927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:40.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.418+0000 7f21927fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:12:40.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.418+0000 7f21927fc640 1 -- 172.21.15.18:0/2650913168 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f219c1427a0 con 0x7f219c13e1e0 2024-03-29T11:12:40.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.418+0000 7f21927fc640 10 monclient: _finish_auth 0 2024-03-29T11:12:40.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.418+0000 7f21927fc640 10 monclient: _check_auth_tickets 2024-03-29T11:12:40.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.418+0000 7f21927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:40.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.418+0000 7f21a4177640 5 monclient: authenticate success, global_id 4382 2024-03-29T11:12:40.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.418+0000 7f21a4177640 10 monclient: _renew_subs 2024-03-29T11:12:40.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.418+0000 7f21a4177640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:12:40.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.418+0000 7f21a4177640 1 -- 172.21.15.18:0/2650913168 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f219c140980 con 0x7f219c13e1e0 2024-03-29T11:12:40.399 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.418+0000 7f21a4177640 10 client.4382.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:40.400 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.418+0000 7f21a4177640 10 monclient: _renew_subs 2024-03-29T11:12:40.400 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.418+0000 7f21a4177640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:12:40.400 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.418+0000 7f21a4177640 1 -- 172.21.15.18:0/2650913168 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f219c136000 con 0x7f219c13e1e0 2024-03-29T11:12:40.400 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.418+0000 7f21a4177640 1 librados: init done 2024-03-29T11:12:40.401 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.420+0000 7f2187fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:40.401 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.420+0000 7f2187fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:40.402 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.420+0000 7f2187fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:12:40.402 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.420+0000 7f2187fff640 1 -- 172.21.15.18:0/2650913168 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f219c142470 con 0x7f219c13e1e0 2024-03-29T11:12:40.402 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.420+0000 7f21927fc640 1 -- 172.21.15.18:0/2650913168 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2194003970 con 0x7f219c13e1e0 2024-03-29T11:12:40.402 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.420+0000 7f21927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:40.402 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.420+0000 7f2193fff640 4 set_mon_vals no callback set 2024-03-29T11:12:40.402 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.421+0000 7f21927fc640 1 -- 172.21.15.18:0/2650913168 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f2194003d00 con 0x7f219c13e1e0 2024-03-29T11:12:40.402 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.421+0000 7f21927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:40.402 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.421+0000 7f21927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:40.402 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.421+0000 7f21927fc640 10 monclient: dump: 2024-03-29T11:12:40.402 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:12:40.402 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:40.402 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:40.402 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:40.403 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:40.403 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:40.403 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:12:40.403 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:12:40.403 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:12:40.403 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:40.403 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.421+0000 7f21927fc640 1 -- 172.21.15.18:0/2650913168 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f2194003f20 con 0x7f219c13e1e0 2024-03-29T11:12:40.403 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.422+0000 7f21927fc640 1 -- 172.21.15.18:0/2650913168 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f2194084b70 con 0x7f219c13e1e0 2024-03-29T11:12:40.403 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.422+0000 7f21927fc640 10 client.4382.objecter ms_dispatch 0x7f219c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:12:40.403 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.422+0000 7f21927fc640 3 client.4382.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:12:40.403 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.422+0000 7f21927fc640 3 client.4382.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:12:40.403 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.422+0000 7f21927fc640 20 client.4382.objecter dump_active .. 0 homeless 2024-03-29T11:12:40.403 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.422+0000 7f21a1f13640 10 monclient: get_auth_request con 0x7f218006dbf0 auth_method 0 2024-03-29T11:12:40.403 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.423+0000 7f21927fc640 10 client.4382.objecter ms_handle_connect 0x7f218006dbf0 2024-03-29T11:12:40.407 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.429+0000 7f21927fc640 1 -- 172.21.15.18:0/2650913168 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2194050450 con 0x7f219c13e1e0 2024-03-29T11:12:40.407 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.429+0000 7f21927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:40.407 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.429+0000 7f21927fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:40.736 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.758+0000 7f2187fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:40.736 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.758+0000 7f2187fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:40.737 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.758+0000 7f2187fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:12:40.737 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.758+0000 7f2187fff640 1 -- 172.21.15.18:0/2650913168 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f219c075f10 con 0x7f219c13e1e0 2024-03-29T11:12:40.737 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.759+0000 7f21927fc640 1 -- 172.21.15.18:0/2650913168 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f2194052380 con 0x7f219c13e1e0 2024-03-29T11:12:40.737 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.759+0000 7f21927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:40.737 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.759+0000 7f21927fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:40.738 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:12:40.738 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:40.742 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.763+0000 7f2187fff640 10 librados: watch_flush enter 2024-03-29T11:12:40.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.763+0000 7f2187fff640 10 librados: watch_flush exit 2024-03-29T11:12:40.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.763+0000 7f2187fff640 20 client.4382.objecter shutdown clearing up homeless session... 2024-03-29T11:12:40.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.763+0000 7f2187fff640 10 client.4382.objecter successfully canceled tick 2024-03-29T11:12:40.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.764+0000 7f2187fff640 1 -- 172.21.15.18:0/2650913168 >> v1:172.21.15.70:6816/3101871562 conn(0x7f218006dbf0 legacy=0x7f21800700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:40.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.764+0000 7f2187fff640 10 monclient: shutdown 2024-03-29T11:12:40.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.764+0000 7f2187fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:40.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.764+0000 7f2187fff640 1 -- 172.21.15.18:0/2650913168 >> v1:172.21.15.18:6790/0 conn(0x7f219c13e1e0 legacy=0x7f219c1ee920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:40.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.764+0000 7f2187fff640 1 -- 172.21.15.18:0/2650913168 shutdown_connections 2024-03-29T11:12:40.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.764+0000 7f2187fff640 1 -- 172.21.15.18:0/2650913168 >> 172.21.15.18:0/2650913168 conn(0x7f219c06b660 msgr2=0x7f219c06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:40.745 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.765+0000 7f2187fff640 1 -- 172.21.15.18:0/2650913168 shutdown_connections 2024-03-29T11:12:40.745 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.766+0000 7f2187fff640 1 -- 172.21.15.18:0/2650913168 wait complete. 2024-03-29T11:12:40.745 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:40.766+0000 7f2187fff640 1 librados: shutdown 2024-03-29T11:12:40.762 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:12:47.571 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:47.705 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.727+0000 7effd5463640 10 monclient: get_monmap_and_config 2024-03-29T11:12:47.706 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.727+0000 7effd5463640 10 monclient: build_initial_monmap 2024-03-29T11:12:47.706 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.727+0000 7effd5463640 10 monclient: monmap: 2024-03-29T11:12:47.706 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:12:47.706 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:47.706 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:12:47.728851+0000 2024-03-29T11:12:47.706 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:12:47.728851+0000 2024-03-29T11:12:47.706 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:47.706 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:47.706 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:12:47.706 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:12:47.706 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:12:47.706 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:47.707 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.728+0000 7effd5463640 1 Processor -- start 2024-03-29T11:12:47.707 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.728+0000 7effd5463640 1 -- start start 2024-03-29T11:12:47.707 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.728+0000 7effd5463640 10 monclient: init 2024-03-29T11:12:47.708 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.729+0000 7effd5463640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:47.708 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.729+0000 7effd5463640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:12:47.708 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.729+0000 7effd5463640 10 monclient(hunting): picked mon.noname-a con 0x7effd006f050 addr v1:172.21.15.18:6789/0 2024-03-29T11:12:47.708 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.729+0000 7effd5463640 10 monclient(hunting): picked mon.noname-b con 0x7effd0070240 addr v1:172.21.15.18:6790/0 2024-03-29T11:12:47.708 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.729+0000 7effd5463640 10 monclient(hunting): picked mon.noname-c con 0x7effd006f430 addr v1:172.21.15.70:6789/0 2024-03-29T11:12:47.708 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.729+0000 7effd5463640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effd013f400 con 0x7effd006f050 2024-03-29T11:12:47.708 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.729+0000 7effd5463640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effd01429c0 con 0x7effd006f430 2024-03-29T11:12:47.708 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.729+0000 7effd5463640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effd006ff80 con 0x7effd0070240 2024-03-29T11:12:47.708 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.729+0000 7effd5463640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:47.708 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.729+0000 7effd5463640 10 monclient(hunting): authenticate will time out at 994.001767s 2024-03-29T11:12:47.708 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.729+0000 7effcf7fe640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7effd006f430 0x7effd014a630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.18:56392/0 (socket says 172.21.15.18:56392) 2024-03-29T11:12:47.708 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.730+0000 7effcf7fe640 1 -- 172.21.15.18:0/1713318752 learned_addr learned my addr 172.21.15.18:0/1713318752 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:12:47.709 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.730+0000 7effcdffb640 1 -- 172.21.15.18:0/1713318752 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3605803598 0 0) 0x7effd006ff80 con 0x7effd0070240 2024-03-29T11:12:47.709 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.730+0000 7effcdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:12:47.709 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.730+0000 7effcdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:47.709 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.730+0000 7effcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:47.709 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.730+0000 7effcdffb640 10 monclient(hunting): my global_id is 4385 2024-03-29T11:12:47.709 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.730+0000 7effcdffb640 1 -- 172.21.15.18:0/1713318752 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effb8005730 con 0x7effd0070240 2024-03-29T11:12:47.709 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.731+0000 7effcdffb640 1 -- 172.21.15.18:0/1713318752 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2355597957 0 0) 0x7effb8005730 con 0x7effd0070240 2024-03-29T11:12:47.709 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.731+0000 7effcdffb640 1 -- 172.21.15.18:0/1713318752 >> v1:172.21.15.70:6789/0 conn(0x7effd006f430 legacy=0x7effd014a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:47.709 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.731+0000 7effcdffb640 1 -- 172.21.15.18:0/1713318752 >> v1:172.21.15.18:6789/0 conn(0x7effd006f050 legacy=0x7effd01422b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:12:47.714 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.733+0000 7effcdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:12:47.714 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.733+0000 7effcdffb640 1 monclient: found mon.noname-b 2024-03-29T11:12:47.715 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.733+0000 7effcdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:47.715 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.733+0000 7effcdffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:12:47.715 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.733+0000 7effcdffb640 1 -- 172.21.15.18:0/1713318752 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd00aef70 con 0x7effd0070240 2024-03-29T11:12:47.715 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.733+0000 7effcdffb640 10 monclient: _finish_auth 0 2024-03-29T11:12:47.715 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.733+0000 7effcdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:12:47.715 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.733+0000 7effcdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:47.715 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.733+0000 7effcdffb640 1 -- 172.21.15.18:0/1713318752 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7effc4001690 con 0x7effd0070240 2024-03-29T11:12:47.715 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.733+0000 7effcdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:47.715 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.733+0000 7effcdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:47.715 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.733+0000 7effcdffb640 10 monclient: dump: 2024-03-29T11:12:47.715 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:12:47.715 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:47.715 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:47.715 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:47.717 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:47.717 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:47.717 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:12:47.717 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:12:47.717 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:12:47.717 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:47.717 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.733+0000 7effd5463640 5 monclient: authenticate success, global_id 4385 2024-03-29T11:12:47.717 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.733+0000 7effd5463640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:47.717 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.733+0000 7effcdffb640 1 -- 172.21.15.18:0/1713318752 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7effc4001860 con 0x7effd0070240 2024-03-29T11:12:47.718 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.733+0000 7effcdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:47.718 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.733+0000 7effcdffb640 1 -- 172.21.15.18:0/1713318752 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7effc4001cb0 con 0x7effd0070240 2024-03-29T11:12:47.718 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.733+0000 7effcdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:47.718 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.733+0000 7effcdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:47.718 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.733+0000 7effcdffb640 10 monclient: dump: 2024-03-29T11:12:47.718 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:12:47.718 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:47.718 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:47.718 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:47.718 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:47.718 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:47.718 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:12:47.718 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:12:47.718 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:12:47.719 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:47.719 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.733+0000 7effd5463640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:47.719 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.733+0000 7effd5463640 4 set_mon_vals no callback set 2024-03-29T11:12:47.719 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.734+0000 7effd5463640 10 monclient: shutdown 2024-03-29T11:12:47.719 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.734+0000 7effd5463640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:47.719 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.734+0000 7effd5463640 1 -- 172.21.15.18:0/1713318752 >> v1:172.21.15.18:6790/0 conn(0x7effd0070240 legacy=0x7effd00ae860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:47.719 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.735+0000 7effd5463640 1 -- 172.21.15.18:0/1713318752 shutdown_connections 2024-03-29T11:12:47.719 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.735+0000 7effd5463640 1 -- 172.21.15.18:0/1713318752 >> 172.21.15.18:0/1713318752 conn(0x7effd00b4930 msgr2=0x7effd00b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:47.719 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.735+0000 7effd5463640 1 -- 172.21.15.18:0/1713318752 shutdown_connections 2024-03-29T11:12:47.719 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.735+0000 7effd5463640 1 -- 172.21.15.18:0/1713318752 wait complete. 2024-03-29T11:12:47.719 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.736+0000 7effd5463640 10 monclient: build_initial_monmap 2024-03-29T11:12:47.719 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.736+0000 7effd5463640 10 monclient: monmap: 2024-03-29T11:12:47.719 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:12:47.719 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:47.719 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:12:47.720 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:12:47.720 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:47.720 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:47.720 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:12:47.720 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:12:47.720 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:12:47.720 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:47.720 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.736+0000 7effd5463640 1 librados: starting msgr at 2024-03-29T11:12:47.720 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.736+0000 7effd5463640 1 librados: starting objecter 2024-03-29T11:12:47.720 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.736+0000 7effd5463640 1 Processor -- start 2024-03-29T11:12:47.720 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.737+0000 7effd5463640 1 -- start start 2024-03-29T11:12:47.720 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.737+0000 7effd5463640 1 librados: setting wanted keys 2024-03-29T11:12:47.720 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.737+0000 7effd5463640 1 librados: calling monclient init 2024-03-29T11:12:47.720 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.737+0000 7effd5463640 10 monclient: init 2024-03-29T11:12:47.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.737+0000 7effd5463640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:47.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.737+0000 7effd5463640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:12:47.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.737+0000 7effd5463640 10 monclient(hunting): picked mon.noname-b con 0x7effd006f050 addr v1:172.21.15.70:6789/0 2024-03-29T11:12:47.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.737+0000 7effd5463640 10 monclient(hunting): picked mon.noname-c con 0x7effd006f430 addr v1:172.21.15.18:6790/0 2024-03-29T11:12:47.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.738+0000 7effceffd640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7effd006f050 0x7effd01d62d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.18:57302/0 (socket says 172.21.15.18:57302) 2024-03-29T11:12:47.732 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.738+0000 7effceffd640 1 -- 172.21.15.18:0/2778140328 learned_addr learned my addr 172.21.15.18:0/2778140328 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:12:47.732 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.738+0000 7effd5463640 10 monclient(hunting): picked mon.noname-a con 0x7effd0070240 addr v1:172.21.15.18:6789/0 2024-03-29T11:12:47.732 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.738+0000 7effd5463640 1 -- 172.21.15.18:0/2778140328 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effd00ad500 con 0x7effd0070240 2024-03-29T11:12:47.732 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.738+0000 7effd5463640 1 -- 172.21.15.18:0/2778140328 --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effd00ae460 con 0x7effd006f050 2024-03-29T11:12:47.732 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.738+0000 7effd5463640 1 -- 172.21.15.18:0/2778140328 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effd013bb40 con 0x7effd006f430 2024-03-29T11:12:47.732 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.738+0000 7effd5463640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:47.732 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.738+0000 7effd5463640 10 monclient(hunting): authenticate will time out at 994.010427s 2024-03-29T11:12:47.732 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.738+0000 7effb77fe640 10 client.?.objecter ms_handle_connect 0x7effd006f050 2024-03-29T11:12:47.732 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.738+0000 7effb77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:47.732 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.738+0000 7effb77fe640 1 -- 172.21.15.18:0/2778140328 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1326959607 0 0) 0x7effd00ae460 con 0x7effd006f050 2024-03-29T11:12:47.732 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.738+0000 7effb77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:12:47.732 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.738+0000 7effb77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:47.732 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.738+0000 7effb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:47.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.738+0000 7effb77fe640 10 monclient(hunting): my global_id is 4354 2024-03-29T11:12:47.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.738+0000 7effb77fe640 1 -- 172.21.15.18:0/2778140328 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effbc004eb0 con 0x7effd006f050 2024-03-29T11:12:47.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.739+0000 7effb77fe640 10 client.?.objecter ms_handle_connect 0x7effd006f430 2024-03-29T11:12:47.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.739+0000 7effb77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:47.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.739+0000 7effb77fe640 1 -- 172.21.15.18:0/2778140328 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2797248585 0 0) 0x7effd013bb40 con 0x7effd006f430 2024-03-29T11:12:47.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.739+0000 7effb77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:12:47.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.739+0000 7effb77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:47.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.739+0000 7effb77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:47.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.739+0000 7effb77fe640 10 monclient(hunting): my global_id is 4388 2024-03-29T11:12:47.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.739+0000 7effb77fe640 1 -- 172.21.15.18:0/2778140328 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effd00ae460 con 0x7effd006f430 2024-03-29T11:12:47.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.739+0000 7effb77fe640 1 -- 172.21.15.18:0/2778140328 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 605149098 0 0) 0x7effbc004eb0 con 0x7effd006f050 2024-03-29T11:12:47.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.740+0000 7effb77fe640 1 -- 172.21.15.18:0/2778140328 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7effd013bb40 con 0x7effd006f050 2024-03-29T11:12:47.734 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.740+0000 7effb77fe640 1 -- 172.21.15.18:0/2778140328 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7effc0001960 con 0x7effd006f050 2024-03-29T11:12:47.734 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.740+0000 7effb77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:47.734 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.740+0000 7effb77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:47.734 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.740+0000 7effb77fe640 10 monclient(hunting): dump: 2024-03-29T11:12:47.734 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:12:47.734 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:47.734 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:47.734 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:47.734 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:47.734 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:47.734 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:12:47.734 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:12:47.735 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:12:47.735 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:47.735 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.740+0000 7effb77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:47.735 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.740+0000 7effb77fe640 1 -- 172.21.15.18:0/2778140328 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4226145266 0 0) 0x7effd00ae460 con 0x7effd006f430 2024-03-29T11:12:47.735 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.740+0000 7effb77fe640 1 -- 172.21.15.18:0/2778140328 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7effbc004eb0 con 0x7effd006f430 2024-03-29T11:12:47.735 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.740+0000 7effb77fe640 1 -- 172.21.15.18:0/2778140328 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7effc4001b70 con 0x7effd006f430 2024-03-29T11:12:47.736 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.740+0000 7effb77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:47.736 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.740+0000 7effb77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:47.736 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.740+0000 7effb77fe640 10 monclient(hunting): dump: 2024-03-29T11:12:47.736 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:12:47.736 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:47.736 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:47.736 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:47.736 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:47.736 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:47.736 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:12:47.736 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:12:47.737 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:12:47.737 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:47.737 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.740+0000 7effb77fe640 1 -- 172.21.15.18:0/2778140328 <== mon.1 v1:172.21.15.70:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2798430523 0 0) 0x7effd013bb40 con 0x7effd006f050 2024-03-29T11:12:47.737 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.740+0000 7effb77fe640 1 -- 172.21.15.18:0/2778140328 >> v1:172.21.15.18:6790/0 conn(0x7effd006f430 legacy=0x7effd013ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:47.737 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.740+0000 7effb77fe640 1 -- 172.21.15.18:0/2778140328 >> v1:172.21.15.18:6789/0 conn(0x7effd0070240 legacy=0x7effd01de610 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:12:47.737 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.741+0000 7effb77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:12:47.737 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.741+0000 7effb77fe640 1 monclient: found mon.b 2024-03-29T11:12:47.737 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.741+0000 7effb77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:47.737 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.741+0000 7effb77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:12:47.737 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.741+0000 7effb77fe640 1 -- 172.21.15.18:0/2778140328 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd013bcd0 con 0x7effd006f050 2024-03-29T11:12:47.737 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.742+0000 7effb77fe640 10 monclient: _finish_auth 0 2024-03-29T11:12:47.737 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.742+0000 7effb77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:12:47.738 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.742+0000 7effb77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:47.738 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.742+0000 7effd5463640 5 monclient: authenticate success, global_id 4354 2024-03-29T11:12:47.738 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.742+0000 7effd5463640 10 monclient: _renew_subs 2024-03-29T11:12:47.738 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.742+0000 7effd5463640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:12:47.738 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.742+0000 7effd5463640 1 -- 172.21.15.18:0/2778140328 --> v1:172.21.15.70:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effd01c9a70 con 0x7effd006f050 2024-03-29T11:12:47.738 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.742+0000 7effb77fe640 1 -- 172.21.15.18:0/2778140328 <== mon.1 v1:172.21.15.70:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7effc00035f0 con 0x7effd006f050 2024-03-29T11:12:47.738 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.742+0000 7effd5463640 10 client.4354.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:47.738 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.742+0000 7effd5463640 10 monclient: _renew_subs 2024-03-29T11:12:47.738 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.742+0000 7effd5463640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:12:47.738 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.742+0000 7effd5463640 1 -- 172.21.15.18:0/2778140328 --> v1:172.21.15.70:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7effd01d6bc0 con 0x7effd006f050 2024-03-29T11:12:47.738 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.742+0000 7effd5463640 1 librados: init done 2024-03-29T11:12:47.738 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.742+0000 7effb77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:47.739 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.742+0000 7effcd7fa640 4 set_mon_vals no callback set 2024-03-29T11:12:47.739 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.743+0000 7effb77fe640 1 -- 172.21.15.18:0/2778140328 <== mon.1 v1:172.21.15.70:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7effc0003940 con 0x7effd006f050 2024-03-29T11:12:47.739 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.743+0000 7effb77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:47.739 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.743+0000 7effb77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:47.739 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.743+0000 7effb77fe640 10 monclient: dump: 2024-03-29T11:12:47.739 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:12:47.739 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:47.739 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:47.739 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:47.739 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:47.739 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:47.739 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:12:47.739 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:12:47.740 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:12:47.740 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:47.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.744+0000 7effd5463640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:47.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.745+0000 7effd5463640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:47.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.745+0000 7effd5463640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:12:47.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.745+0000 7effd5463640 1 -- 172.21.15.18:0/2778140328 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effb0005080 con 0x7effd006f050 2024-03-29T11:12:47.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.745+0000 7effb77fe640 1 -- 172.21.15.18:0/2778140328 <== mon.1 v1:172.21.15.70:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7effc0003aa0 con 0x7effd006f050 2024-03-29T11:12:47.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.746+0000 7effce7fc640 10 monclient: get_auth_request con 0x7effbc06ddb0 auth_method 0 2024-03-29T11:12:47.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.746+0000 7effb77fe640 1 -- 172.21.15.18:0/2778140328 <== mon.1 v1:172.21.15.70:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7effc00849f0 con 0x7effd006f050 2024-03-29T11:12:47.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.746+0000 7effb77fe640 10 client.4354.objecter ms_dispatch 0x7effd0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:12:47.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.746+0000 7effb77fe640 3 client.4354.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:12:47.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.746+0000 7effb77fe640 3 client.4354.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:12:47.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.746+0000 7effb77fe640 20 client.4354.objecter dump_active .. 0 homeless 2024-03-29T11:12:47.741 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.746+0000 7effb77fe640 10 client.4354.objecter ms_handle_connect 0x7effbc06ddb0 2024-03-29T11:12:47.741 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.754+0000 7effb77fe640 1 -- 172.21.15.18:0/2778140328 <== mon.1 v1:172.21.15.70:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7effc00502d0 con 0x7effd006f050 2024-03-29T11:12:47.741 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.755+0000 7effb77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:47.741 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:47.755+0000 7effb77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:48.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:48.061+0000 7effd5463640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:48.040 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:48.061+0000 7effd5463640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:48.040 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:48.061+0000 7effd5463640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:12:48.040 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:48.062+0000 7effd5463640 1 -- 172.21.15.18:0/2778140328 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7effb0005330 con 0x7effd006f050 2024-03-29T11:12:48.040 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:48.063+0000 7effb77fe640 1 -- 172.21.15.18:0/2778140328 <== mon.1 v1:172.21.15.70:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7effc0052200 con 0x7effd006f050 2024-03-29T11:12:48.041 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:48.063+0000 7effb77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:48.041 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:48.063+0000 7effb77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:48.041 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:12:48.041 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:48.045 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:48.065+0000 7effd5463640 10 librados: watch_flush enter 2024-03-29T11:12:48.045 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:48.065+0000 7effd5463640 10 librados: watch_flush exit 2024-03-29T11:12:48.045 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:48.065+0000 7effd5463640 20 client.4354.objecter shutdown clearing up homeless session... 2024-03-29T11:12:48.045 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:48.065+0000 7effd5463640 10 client.4354.objecter successfully canceled tick 2024-03-29T11:12:48.045 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:48.065+0000 7effd5463640 1 -- 172.21.15.18:0/2778140328 >> v1:172.21.15.70:6816/3101871562 conn(0x7effbc06ddb0 legacy=0x7effbc070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:48.045 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:48.065+0000 7effd5463640 10 monclient: shutdown 2024-03-29T11:12:48.045 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:48.065+0000 7effd5463640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:48.045 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:48.065+0000 7effd5463640 1 -- 172.21.15.18:0/2778140328 >> v1:172.21.15.70:6789/0 conn(0x7effd006f050 legacy=0x7effd01d62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:48.045 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:48.065+0000 7effd5463640 1 -- 172.21.15.18:0/2778140328 shutdown_connections 2024-03-29T11:12:48.045 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:48.065+0000 7effd5463640 1 -- 172.21.15.18:0/2778140328 >> 172.21.15.18:0/2778140328 conn(0x7effd00b4930 msgr2=0x7effd006b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:48.045 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:48.066+0000 7effd5463640 1 -- 172.21.15.18:0/2778140328 shutdown_connections 2024-03-29T11:12:48.045 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:48.066+0000 7effd5463640 1 -- 172.21.15.18:0/2778140328 wait complete. 2024-03-29T11:12:48.045 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:48.066+0000 7effd5463640 1 librados: shutdown 2024-03-29T11:12:48.059 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:12:54.868 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:12:55.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.043+0000 7f4f38b40640 10 monclient: get_monmap_and_config 2024-03-29T11:12:55.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.043+0000 7f4f38b40640 10 monclient: build_initial_monmap 2024-03-29T11:12:55.021 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.043+0000 7f4f38b40640 10 monclient: monmap: 2024-03-29T11:12:55.021 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:12:55.021 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:55.022 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:12:55.044415+0000 2024-03-29T11:12:55.022 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:12:55.044415+0000 2024-03-29T11:12:55.022 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:55.022 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:55.022 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:12:55.022 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:12:55.022 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:12:55.022 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:55.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.045+0000 7f4f38b40640 1 Processor -- start 2024-03-29T11:12:55.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.045+0000 7f4f38b40640 1 -- start start 2024-03-29T11:12:55.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.045+0000 7f4f38b40640 10 monclient: init 2024-03-29T11:12:55.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.047+0000 7f4f38b40640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:55.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.047+0000 7f4f38b40640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:12:55.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.047+0000 7f4f38b40640 10 monclient(hunting): picked mon.noname-a con 0x7f4f341422b0 addr v1:172.21.15.18:6789/0 2024-03-29T11:12:55.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.047+0000 7f4f38b40640 10 monclient(hunting): picked mon.noname-c con 0x7f4f34142690 addr v1:172.21.15.70:6789/0 2024-03-29T11:12:55.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.047+0000 7f4f38b40640 10 monclient(hunting): picked mon.noname-b con 0x7f4f340b17d0 addr v1:172.21.15.18:6790/0 2024-03-29T11:12:55.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.047+0000 7f4f38b40640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f34142c80 con 0x7f4f341422b0 2024-03-29T11:12:55.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.047+0000 7f4f38b40640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f340aae40 con 0x7f4f34142690 2024-03-29T11:12:55.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.047+0000 7f4f38b40640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f3413fbd0 con 0x7f4f340b17d0 2024-03-29T11:12:55.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.047+0000 7f4f38b40640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:55.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.047+0000 7f4f38b40640 10 monclient(hunting): authenticate will time out at 1001.319398s 2024-03-29T11:12:55.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.048+0000 7f4f32d9d640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f4f340b17d0 0x7f4f340b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:59838/0 (socket says 172.21.15.18:59838) 2024-03-29T11:12:55.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.048+0000 7f4f32d9d640 1 -- 172.21.15.18:0/422120954 learned_addr learned my addr 172.21.15.18:0/422120954 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:12:55.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.049+0000 7f4f3159a640 1 -- 172.21.15.18:0/422120954 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2470783996 0 0) 0x7f4f3413fbd0 con 0x7f4f340b17d0 2024-03-29T11:12:55.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.049+0000 7f4f3159a640 10 monclient(hunting): _negotiate 2024-03-29T11:12:55.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.049+0000 7f4f3159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:55.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.049+0000 7f4f3159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:55.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.049+0000 7f4f3159a640 10 monclient(hunting): my global_id is 4391 2024-03-29T11:12:55.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.049+0000 7f4f3159a640 1 -- 172.21.15.18:0/422120954 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f1c003520 con 0x7f4f340b17d0 2024-03-29T11:12:55.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.049+0000 7f4f3159a640 1 -- 172.21.15.18:0/422120954 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3357205436 0 0) 0x7f4f1c003520 con 0x7f4f340b17d0 2024-03-29T11:12:55.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.049+0000 7f4f3159a640 1 -- 172.21.15.18:0/422120954 >> v1:172.21.15.70:6789/0 conn(0x7f4f34142690 legacy=0x7f4f340b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:12:55.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.051+0000 7f4f3159a640 1 -- 172.21.15.18:0/422120954 >> v1:172.21.15.18:6789/0 conn(0x7f4f341422b0 legacy=0x7f4f3413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:12:55.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.051+0000 7f4f3159a640 10 monclient: _finish_hunting 0 2024-03-29T11:12:55.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.051+0000 7f4f3159a640 1 monclient: found mon.noname-b 2024-03-29T11:12:55.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.051+0000 7f4f3159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:55.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.051+0000 7f4f3159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:12:55.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.051+0000 7f4f3159a640 1 -- 172.21.15.18:0/422120954 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f34151940 con 0x7f4f340b17d0 2024-03-29T11:12:55.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.051+0000 7f4f3159a640 10 monclient: _finish_auth 0 2024-03-29T11:12:55.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.051+0000 7f4f3159a640 10 monclient: _check_auth_tickets 2024-03-29T11:12:55.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.051+0000 7f4f3159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:55.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.051+0000 7f4f3159a640 1 -- 172.21.15.18:0/422120954 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f4f24001690 con 0x7f4f340b17d0 2024-03-29T11:12:55.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.051+0000 7f4f3159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:55.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.051+0000 7f4f3159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:12:55.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.051+0000 7f4f3159a640 10 monclient: dump: 2024-03-29T11:12:55.029 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:12:55.029 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:55.029 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:55.030 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:55.030 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:55.030 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:55.030 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:12:55.030 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:12:55.030 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:12:55.030 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:55.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.053+0000 7f4f38b40640 5 monclient: authenticate success, global_id 4391 2024-03-29T11:12:55.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.053+0000 7f4f38b40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:12:55.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.053+0000 7f4f3159a640 1 -- 172.21.15.18:0/422120954 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f240017f0 con 0x7f4f340b17d0 2024-03-29T11:12:55.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.053+0000 7f4f3159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:55.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.053+0000 7f4f38b40640 10 monclient: get_monmap_and_config success 2024-03-29T11:12:55.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.053+0000 7f4f38b40640 4 set_mon_vals no callback set 2024-03-29T11:12:55.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.054+0000 7f4f3159a640 1 -- 172.21.15.18:0/422120954 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f4f24001c40 con 0x7f4f340b17d0 2024-03-29T11:12:55.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.054+0000 7f4f3159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:55.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.054+0000 7f4f3159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:55.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.054+0000 7f4f3159a640 10 monclient: dump: 2024-03-29T11:12:55.032 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:12:55.033 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:55.033 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:55.033 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:55.033 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:55.033 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:55.033 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:12:55.033 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:12:55.033 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:12:55.033 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:55.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.054+0000 7f4f38b40640 10 monclient: shutdown 2024-03-29T11:12:55.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.054+0000 7f4f38b40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:55.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.054+0000 7f4f38b40640 1 -- 172.21.15.18:0/422120954 >> v1:172.21.15.18:6790/0 conn(0x7f4f340b17d0 legacy=0x7f4f340b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:55.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.054+0000 7f4f38b40640 1 -- 172.21.15.18:0/422120954 shutdown_connections 2024-03-29T11:12:55.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.054+0000 7f4f38b40640 1 -- 172.21.15.18:0/422120954 >> 172.21.15.18:0/422120954 conn(0x7f4f34066a00 msgr2=0x7f4f34066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:55.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.054+0000 7f4f38b40640 1 -- 172.21.15.18:0/422120954 shutdown_connections 2024-03-29T11:12:55.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.055+0000 7f4f38b40640 1 -- 172.21.15.18:0/422120954 wait complete. 2024-03-29T11:12:55.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.056+0000 7f4f38b40640 10 monclient: build_initial_monmap 2024-03-29T11:12:55.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.056+0000 7f4f38b40640 10 monclient: monmap: 2024-03-29T11:12:55.036 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:12:55.036 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:12:55.036 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:12:55.036 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:12:55.036 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:12:55.036 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:55.036 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:12:55.036 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:12:55.036 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:12:55.037 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:55.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.056+0000 7f4f38b40640 1 librados: starting msgr at 2024-03-29T11:12:55.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.056+0000 7f4f38b40640 1 librados: starting objecter 2024-03-29T11:12:55.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.056+0000 7f4f38b40640 1 Processor -- start 2024-03-29T11:12:55.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.056+0000 7f4f38b40640 1 -- start start 2024-03-29T11:12:55.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.056+0000 7f4f38b40640 1 librados: setting wanted keys 2024-03-29T11:12:55.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.056+0000 7f4f38b40640 1 librados: calling monclient init 2024-03-29T11:12:55.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.056+0000 7f4f38b40640 10 monclient: init 2024-03-29T11:12:55.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.057+0000 7f4f38b40640 10 monclient: _reopen_session rank -1 2024-03-29T11:12:55.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.057+0000 7f4f38b40640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:12:55.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.057+0000 7f4f38b40640 10 monclient(hunting): picked mon.noname-a con 0x7f4f340b17d0 addr v1:172.21.15.18:6789/0 2024-03-29T11:12:55.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.057+0000 7f4f38b40640 10 monclient(hunting): picked mon.noname-c con 0x7f4f341422b0 addr v1:172.21.15.18:6790/0 2024-03-29T11:12:55.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.057+0000 7f4f38b40640 10 monclient(hunting): picked mon.noname-b con 0x7f4f34142690 addr v1:172.21.15.70:6789/0 2024-03-29T11:12:55.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.057+0000 7f4f38b40640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f3406eb10 con 0x7f4f340b17d0 2024-03-29T11:12:55.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.057+0000 7f4f38b40640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f34071ff0 con 0x7f4f34142690 2024-03-29T11:12:55.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.057+0000 7f4f38b40640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f34070e70 con 0x7f4f341422b0 2024-03-29T11:12:55.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.057+0000 7f4f38b40640 10 monclient(hunting): _renew_subs 2024-03-29T11:12:55.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.057+0000 7f4f38b40640 10 monclient(hunting): authenticate will time out at 1001.329276s 2024-03-29T11:12:55.042 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.060+0000 7f4f31d9b640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f4f341422b0 0x7f4f3406efa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:60066/0 (socket says 172.21.15.18:60066) 2024-03-29T11:12:55.042 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.060+0000 7f4f31d9b640 1 -- 172.21.15.18:0/3547636336 learned_addr learned my addr 172.21.15.18:0/3547636336 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:12:55.042 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.060+0000 7f4f22ffd640 10 client.?.objecter ms_handle_connect 0x7f4f341422b0 2024-03-29T11:12:55.042 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.060+0000 7f4f22ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:55.042 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.060+0000 7f4f22ffd640 1 -- 172.21.15.18:0/3547636336 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2325588731 0 0) 0x7f4f34070e70 con 0x7f4f341422b0 2024-03-29T11:12:55.042 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.060+0000 7f4f22ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:55.043 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.060+0000 7f4f22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:55.043 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.060+0000 7f4f22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:55.043 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.061+0000 7f4f22ffd640 10 monclient(hunting): my global_id is 4394 2024-03-29T11:12:55.043 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.061+0000 7f4f22ffd640 1 -- 172.21.15.18:0/3547636336 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f14004eb0 con 0x7f4f341422b0 2024-03-29T11:12:55.043 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.061+0000 7f4f22ffd640 10 client.?.objecter ms_handle_connect 0x7f4f34142690 2024-03-29T11:12:55.043 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.061+0000 7f4f22ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:12:55.043 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.061+0000 7f4f22ffd640 1 -- 172.21.15.18:0/3547636336 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1226152538 0 0) 0x7f4f34071ff0 con 0x7f4f34142690 2024-03-29T11:12:55.043 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.061+0000 7f4f22ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:12:55.043 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.061+0000 7f4f22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:12:55.043 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.061+0000 7f4f22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:12:55.043 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.061+0000 7f4f22ffd640 10 monclient(hunting): my global_id is 4357 2024-03-29T11:12:55.043 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.061+0000 7f4f22ffd640 1 -- 172.21.15.18:0/3547636336 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f34070e70 con 0x7f4f34142690 2024-03-29T11:12:55.043 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.061+0000 7f4f22ffd640 1 -- 172.21.15.18:0/3547636336 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1564775824 0 0) 0x7f4f14004eb0 con 0x7f4f341422b0 2024-03-29T11:12:55.043 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.061+0000 7f4f22ffd640 1 -- 172.21.15.18:0/3547636336 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f34071ff0 con 0x7f4f341422b0 2024-03-29T11:12:55.043 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.062+0000 7f4f22ffd640 1 -- 172.21.15.18:0/3547636336 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f4f2c001b50 con 0x7f4f341422b0 2024-03-29T11:12:55.044 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.062+0000 7f4f22ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:55.044 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.062+0000 7f4f22ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:12:55.044 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.062+0000 7f4f22ffd640 10 monclient(hunting): dump: 2024-03-29T11:12:55.044 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:12:55.044 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:55.044 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:55.044 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:55.044 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:55.044 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:55.044 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:12:55.044 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:12:55.044 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:12:55.044 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:55.044 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.062+0000 7f4f22ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:12:55.044 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.062+0000 7f4f22ffd640 1 -- 172.21.15.18:0/3547636336 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2955452637 0 0) 0x7f4f34070e70 con 0x7f4f34142690 2024-03-29T11:12:55.045 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.062+0000 7f4f22ffd640 1 -- 172.21.15.18:0/3547636336 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f14004eb0 con 0x7f4f34142690 2024-03-29T11:12:55.045 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.062+0000 7f4f22ffd640 1 -- 172.21.15.18:0/3547636336 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f4f24001b00 con 0x7f4f34142690 2024-03-29T11:12:55.045 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.062+0000 7f4f22ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:55.045 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.062+0000 7f4f22ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:12:55.045 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.062+0000 7f4f22ffd640 10 monclient(hunting): dump: 2024-03-29T11:12:55.045 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:12:55.045 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:55.045 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:55.045 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:55.045 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:55.045 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:55.045 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:12:55.045 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:12:55.045 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:12:55.046 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:55.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.062+0000 7f4f22ffd640 1 -- 172.21.15.18:0/3547636336 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1608824415 0 0) 0x7f4f34071ff0 con 0x7f4f341422b0 2024-03-29T11:12:55.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.062+0000 7f4f22ffd640 1 -- 172.21.15.18:0/3547636336 >> v1:172.21.15.70:6789/0 conn(0x7f4f34142690 legacy=0x7f4f3406f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:55.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.062+0000 7f4f22ffd640 1 -- 172.21.15.18:0/3547636336 >> v1:172.21.15.18:6789/0 conn(0x7f4f340b17d0 legacy=0x7f4f340718e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:12:55.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.064+0000 7f4f22ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:12:55.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.064+0000 7f4f22ffd640 1 monclient: found mon.c 2024-03-29T11:12:55.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.064+0000 7f4f22ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:12:55.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.064+0000 7f4f22ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:12:55.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.064+0000 7f4f22ffd640 1 -- 172.21.15.18:0/3547636336 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f34071150 con 0x7f4f341422b0 2024-03-29T11:12:55.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.064+0000 7f4f22ffd640 10 monclient: _finish_auth 0 2024-03-29T11:12:55.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.064+0000 7f4f22ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:12:55.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.064+0000 7f4f22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:12:55.048 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.064+0000 7f4f38b40640 5 monclient: authenticate success, global_id 4394 2024-03-29T11:12:55.048 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.064+0000 7f4f38b40640 10 monclient: _renew_subs 2024-03-29T11:12:55.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.064+0000 7f4f38b40640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:12:55.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.064+0000 7f4f38b40640 1 -- 172.21.15.18:0/3547636336 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f340b0e80 con 0x7f4f341422b0 2024-03-29T11:12:55.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.064+0000 7f4f38b40640 10 client.4394.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:12:55.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.064+0000 7f4f38b40640 10 monclient: _renew_subs 2024-03-29T11:12:55.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.064+0000 7f4f38b40640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:12:55.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.064+0000 7f4f38b40640 1 -- 172.21.15.18:0/3547636336 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f341e7380 con 0x7f4f341422b0 2024-03-29T11:12:55.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.064+0000 7f4f38b40640 1 librados: init done 2024-03-29T11:12:55.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.066+0000 7f4f20ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:55.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.066+0000 7f4f20ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:55.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.066+0000 7f4f20ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:12:55.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.066+0000 7f4f20ff9640 1 -- 172.21.15.18:0/3547636336 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f18005080 con 0x7f4f341422b0 2024-03-29T11:12:55.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.066+0000 7f4f22ffd640 1 -- 172.21.15.18:0/3547636336 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f2c001cb0 con 0x7f4f341422b0 2024-03-29T11:12:55.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.066+0000 7f4f22ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:12:55.049 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.066+0000 7f4f22ffd640 1 -- 172.21.15.18:0/3547636336 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f4f2c004d80 con 0x7f4f341422b0 2024-03-29T11:12:55.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.066+0000 7f4f22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:12:55.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.066+0000 7f4f22ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:12:55.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.066+0000 7f4f22ffd640 10 monclient: dump: 2024-03-29T11:12:55.050 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:12:55.050 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:12:55.050 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:55.050 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:12:55.050 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:12:55.050 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:12:55.050 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:12:55.050 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:12:55.050 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:12:55.051 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:12:55.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.067+0000 7f4f30d99640 4 set_mon_vals no callback set 2024-03-29T11:12:55.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.070+0000 7f4f22ffd640 1 -- 172.21.15.18:0/3547636336 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f4f2c01ad50 con 0x7f4f341422b0 2024-03-29T11:12:55.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.071+0000 7f4f22ffd640 1 -- 172.21.15.18:0/3547636336 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f4f2c084cb0 con 0x7f4f341422b0 2024-03-29T11:12:55.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.071+0000 7f4f22ffd640 10 client.4394.objecter ms_dispatch 0x7f4f34000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:12:55.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.071+0000 7f4f22ffd640 3 client.4394.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:12:55.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.071+0000 7f4f22ffd640 3 client.4394.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:12:55.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.071+0000 7f4f22ffd640 20 client.4394.objecter dump_active .. 0 homeless 2024-03-29T11:12:55.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.071+0000 7f4f3259c640 10 monclient: get_auth_request con 0x7f4f1406de80 auth_method 0 2024-03-29T11:12:55.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.072+0000 7f4f22ffd640 10 client.4394.objecter ms_handle_connect 0x7f4f1406de80 2024-03-29T11:12:55.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.074+0000 7f4f22ffd640 1 -- 172.21.15.18:0/3547636336 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4f2c0503c0 con 0x7f4f341422b0 2024-03-29T11:12:55.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.074+0000 7f4f22ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:12:55.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.074+0000 7f4f22ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:12:55.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.395+0000 7f4f20ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:55.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.395+0000 7f4f20ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:55.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.395+0000 7f4f20ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:12:55.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.395+0000 7f4f20ff9640 1 -- 172.21.15.18:0/3547636336 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f18005330 con 0x7f4f341422b0 2024-03-29T11:12:55.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.396+0000 7f4f22ffd640 1 -- 172.21.15.18:0/3547636336 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f4f2c0524c0 con 0x7f4f341422b0 2024-03-29T11:12:55.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.396+0000 7f4f22ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:12:55.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.396+0000 7f4f22ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:12:55.374 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:12:55.375 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:12:55.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.399+0000 7f4f20ff9640 10 librados: watch_flush enter 2024-03-29T11:12:55.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.399+0000 7f4f20ff9640 10 librados: watch_flush exit 2024-03-29T11:12:55.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.399+0000 7f4f20ff9640 20 client.4394.objecter shutdown clearing up homeless session... 2024-03-29T11:12:55.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.399+0000 7f4f20ff9640 10 client.4394.objecter successfully canceled tick 2024-03-29T11:12:55.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.400+0000 7f4f20ff9640 1 -- 172.21.15.18:0/3547636336 >> v1:172.21.15.70:6816/3101871562 conn(0x7f4f1406de80 legacy=0x7f4f14070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:55.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.400+0000 7f4f20ff9640 10 monclient: shutdown 2024-03-29T11:12:55.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.400+0000 7f4f20ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:12:55.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.400+0000 7f4f20ff9640 1 -- 172.21.15.18:0/3547636336 >> v1:172.21.15.18:6790/0 conn(0x7f4f341422b0 legacy=0x7f4f3406efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:12:55.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.400+0000 7f4f20ff9640 1 -- 172.21.15.18:0/3547636336 shutdown_connections 2024-03-29T11:12:55.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.400+0000 7f4f20ff9640 1 -- 172.21.15.18:0/3547636336 >> 172.21.15.18:0/3547636336 conn(0x7f4f34066a00 msgr2=0x7f4f3414d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:12:55.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.400+0000 7f4f20ff9640 1 -- 172.21.15.18:0/3547636336 shutdown_connections 2024-03-29T11:12:55.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.400+0000 7f4f20ff9640 1 -- 172.21.15.18:0/3547636336 wait complete. 2024-03-29T11:12:55.379 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:12:55.400+0000 7f4f20ff9640 1 librados: shutdown 2024-03-29T11:12:55.393 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:13:02.200 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:13:02.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.371+0000 7f64695c8640 10 monclient: get_monmap_and_config 2024-03-29T11:13:02.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.371+0000 7f64695c8640 10 monclient: build_initial_monmap 2024-03-29T11:13:02.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.371+0000 7f64695c8640 10 monclient: monmap: 2024-03-29T11:13:02.352 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:13:02.352 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:02.352 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:13:02.372048+0000 2024-03-29T11:13:02.352 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:13:02.372048+0000 2024-03-29T11:13:02.352 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:02.352 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:02.352 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:13:02.352 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:13:02.352 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:13:02.353 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:02.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.373+0000 7f64695c8640 1 Processor -- start 2024-03-29T11:13:02.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.373+0000 7f64695c8640 1 -- start start 2024-03-29T11:13:02.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.373+0000 7f64695c8640 10 monclient: init 2024-03-29T11:13:02.355 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.374+0000 7f64695c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:02.355 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.374+0000 7f64695c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:13:02.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.374+0000 7f64695c8640 10 monclient(hunting): picked mon.noname-b con 0x7f645c034850 addr v1:172.21.15.18:6790/0 2024-03-29T11:13:02.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.374+0000 7f64695c8640 10 monclient(hunting): picked mon.noname-c con 0x7f645c0c51c0 addr v1:172.21.15.70:6789/0 2024-03-29T11:13:02.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.374+0000 7f64695c8640 10 monclient(hunting): picked mon.noname-a con 0x7f645c0c7cd0 addr v1:172.21.15.18:6789/0 2024-03-29T11:13:02.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.374+0000 7f64695c8640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f645c029190 con 0x7f645c0c7cd0 2024-03-29T11:13:02.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.374+0000 7f64695c8640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f645c0293e0 con 0x7f645c0c51c0 2024-03-29T11:13:02.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.374+0000 7f64695c8640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f645c0c8340 con 0x7f645c034850 2024-03-29T11:13:02.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.374+0000 7f6463fff640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f645c034850 0x7f645c0c4ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:48460/0 (socket says 172.21.15.18:48460) 2024-03-29T11:13:02.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.374+0000 7f6463fff640 1 -- 172.21.15.18:0/4231844857 learned_addr learned my addr 172.21.15.18:0/4231844857 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:13:02.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.375+0000 7f64695c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:02.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.375+0000 7f64695c8640 10 monclient(hunting): authenticate will time out at 1008.647061s 2024-03-29T11:13:02.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.375+0000 7f6462ffd640 1 -- 172.21.15.18:0/4231844857 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4144493455 0 0) 0x7f645c0c8340 con 0x7f645c034850 2024-03-29T11:13:02.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.375+0000 7f6462ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:13:02.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.375+0000 7f6462ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:02.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.375+0000 7f6462ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:02.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.375+0000 7f6462ffd640 10 monclient(hunting): my global_id is 4397 2024-03-29T11:13:02.358 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.375+0000 7f6462ffd640 1 -- 172.21.15.18:0/4231844857 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6454003520 con 0x7f645c034850 2024-03-29T11:13:02.358 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.375+0000 7f6462ffd640 1 -- 172.21.15.18:0/4231844857 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1743024485 0 0) 0x7f6454003520 con 0x7f645c034850 2024-03-29T11:13:02.358 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.375+0000 7f6462ffd640 1 -- 172.21.15.18:0/4231844857 >> v1:172.21.15.70:6789/0 conn(0x7f645c0c51c0 legacy=0x7f645c029800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:13:02.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.378+0000 7f6462ffd640 1 -- 172.21.15.18:0/4231844857 >> v1:172.21.15.18:6789/0 conn(0x7f645c0c7cd0 legacy=0x7f645c029f10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:13:02.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.380+0000 7f6462ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:13:02.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.380+0000 7f6462ffd640 1 monclient: found mon.noname-b 2024-03-29T11:13:02.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.380+0000 7f6462ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:02.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.380+0000 7f6462ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:13:02.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.380+0000 7f6462ffd640 1 -- 172.21.15.18:0/4231844857 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f645c032980 con 0x7f645c034850 2024-03-29T11:13:02.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.380+0000 7f6462ffd640 10 monclient: _finish_auth 0 2024-03-29T11:13:02.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.380+0000 7f6462ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:13:02.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.380+0000 7f6462ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:02.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.380+0000 7f6462ffd640 1 -- 172.21.15.18:0/4231844857 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f646404f730 con 0x7f645c034850 2024-03-29T11:13:02.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.380+0000 7f6462ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:02.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.380+0000 7f6462ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:13:02.363 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.380+0000 7f6462ffd640 10 monclient: dump: 2024-03-29T11:13:02.363 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:13:02.363 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:02.364 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:02.364 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:02.364 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:02.364 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:02.364 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:13:02.364 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:13:02.364 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:13:02.364 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:02.364 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.380+0000 7f64695c8640 5 monclient: authenticate success, global_id 4397 2024-03-29T11:13:02.365 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.380+0000 7f64695c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:02.365 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.380+0000 7f6462ffd640 1 -- 172.21.15.18:0/4231844857 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64640008d0 con 0x7f645c034850 2024-03-29T11:13:02.365 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.380+0000 7f6462ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:02.365 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.380+0000 7f64695c8640 10 monclient: get_monmap_and_config success 2024-03-29T11:13:02.365 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.380+0000 7f64695c8640 4 set_mon_vals no callback set 2024-03-29T11:13:02.365 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.381+0000 7f6462ffd640 1 -- 172.21.15.18:0/4231844857 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f64640758b0 con 0x7f645c034850 2024-03-29T11:13:02.365 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.381+0000 7f6462ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:02.366 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.381+0000 7f6462ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:13:02.366 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.381+0000 7f6462ffd640 10 monclient: dump: 2024-03-29T11:13:02.366 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:13:02.367 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:02.367 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:02.367 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:02.367 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:02.367 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:02.367 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:13:02.367 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:13:02.368 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:13:02.368 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:02.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.381+0000 7f64695c8640 10 monclient: shutdown 2024-03-29T11:13:02.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.381+0000 7f64695c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:02.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.381+0000 7f64695c8640 1 -- 172.21.15.18:0/4231844857 >> v1:172.21.15.18:6790/0 conn(0x7f645c034850 legacy=0x7f645c0c4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:02.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.383+0000 7f64695c8640 1 -- 172.21.15.18:0/4231844857 shutdown_connections 2024-03-29T11:13:02.369 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.383+0000 7f64695c8640 1 -- 172.21.15.18:0/4231844857 >> 172.21.15.18:0/4231844857 conn(0x7f645c0058f0 msgr2=0x7f645c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:02.369 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.383+0000 7f64695c8640 1 -- 172.21.15.18:0/4231844857 shutdown_connections 2024-03-29T11:13:02.369 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.383+0000 7f64695c8640 1 -- 172.21.15.18:0/4231844857 wait complete. 2024-03-29T11:13:02.380 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.385+0000 7f64695c8640 10 monclient: build_initial_monmap 2024-03-29T11:13:02.380 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.385+0000 7f64695c8640 10 monclient: monmap: 2024-03-29T11:13:02.380 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:13:02.380 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:13:02.380 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:13:02.380 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:13:02.380 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:02.380 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:02.380 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:13:02.380 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:13:02.380 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:13:02.381 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:02.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.385+0000 7f64695c8640 1 librados: starting msgr at 2024-03-29T11:13:02.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.385+0000 7f64695c8640 1 librados: starting objecter 2024-03-29T11:13:02.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.385+0000 7f64695c8640 1 Processor -- start 2024-03-29T11:13:02.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.387+0000 7f64695c8640 1 -- start start 2024-03-29T11:13:02.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.387+0000 7f64695c8640 1 librados: setting wanted keys 2024-03-29T11:13:02.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.387+0000 7f64695c8640 1 librados: calling monclient init 2024-03-29T11:13:02.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.387+0000 7f64695c8640 10 monclient: init 2024-03-29T11:13:02.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.388+0000 7f64695c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:02.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.388+0000 7f64695c8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:13:02.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.388+0000 7f64695c8640 10 monclient(hunting): picked mon.noname-b con 0x7f645c034850 addr v1:172.21.15.70:6789/0 2024-03-29T11:13:02.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.388+0000 7f64695c8640 10 monclient(hunting): picked mon.noname-c con 0x7f645c0c51c0 addr v1:172.21.15.18:6790/0 2024-03-29T11:13:02.382 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.388+0000 7f64695c8640 10 monclient(hunting): picked mon.noname-a con 0x7f645c0c7cd0 addr v1:172.21.15.18:6789/0 2024-03-29T11:13:02.382 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.388+0000 7f64695c8640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f645c0c12f0 con 0x7f645c0c7cd0 2024-03-29T11:13:02.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.388+0000 7f64695c8640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f645c0c04d0 con 0x7f645c034850 2024-03-29T11:13:02.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.388+0000 7f64695c8640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f645c0c8c30 con 0x7f645c0c51c0 2024-03-29T11:13:02.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.388+0000 7f64695c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:02.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.388+0000 7f64695c8640 10 monclient(hunting): authenticate will time out at 1008.660805s 2024-03-29T11:13:02.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.390+0000 7f64637fe640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f645c0c51c0 0x7f645c160960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:49140/0 (socket says 172.21.15.18:49140) 2024-03-29T11:13:02.383 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.390+0000 7f64637fe640 1 -- 172.21.15.18:0/732766033 learned_addr learned my addr 172.21.15.18:0/732766033 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:13:02.384 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.390+0000 7f6460ff9640 10 client.?.objecter ms_handle_connect 0x7f645c0c51c0 2024-03-29T11:13:02.384 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.390+0000 7f6460ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:02.384 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.390+0000 7f6460ff9640 1 -- 172.21.15.18:0/732766033 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 392272418 0 0) 0x7f645c0c8c30 con 0x7f645c0c51c0 2024-03-29T11:13:02.384 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.391+0000 7f6460ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:13:02.384 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.391+0000 7f6460ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:02.384 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.391+0000 7f6460ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:02.384 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.391+0000 7f6460ff9640 10 monclient(hunting): my global_id is 4400 2024-03-29T11:13:02.384 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.391+0000 7f6460ff9640 1 -- 172.21.15.18:0/732766033 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6448004eb0 con 0x7f645c0c51c0 2024-03-29T11:13:02.384 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.391+0000 7f6460ff9640 1 -- 172.21.15.18:0/732766033 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2012994034 0 0) 0x7f6448004eb0 con 0x7f645c0c51c0 2024-03-29T11:13:02.384 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.391+0000 7f6460ff9640 1 -- 172.21.15.18:0/732766033 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f645c0c8c30 con 0x7f645c0c51c0 2024-03-29T11:13:02.384 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.391+0000 7f6460ff9640 1 -- 172.21.15.18:0/732766033 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f6458001870 con 0x7f645c0c51c0 2024-03-29T11:13:02.384 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.391+0000 7f6460ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:02.384 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.391+0000 7f6460ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:13:02.384 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.391+0000 7f6460ff9640 10 monclient(hunting): dump: 2024-03-29T11:13:02.385 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:13:02.385 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:02.385 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:02.385 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:02.385 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:02.385 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:02.385 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:13:02.385 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:13:02.385 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:13:02.385 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:02.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.391+0000 7f6460ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:13:02.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.391+0000 7f6460ff9640 1 -- 172.21.15.18:0/732766033 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 276899311 0 0) 0x7f645c0c8c30 con 0x7f645c0c51c0 2024-03-29T11:13:02.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.391+0000 7f6460ff9640 1 -- 172.21.15.18:0/732766033 >> v1:172.21.15.70:6789/0 conn(0x7f645c034850 legacy=0x7f645c160250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:13:02.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.392+0000 7f6460ff9640 1 -- 172.21.15.18:0/732766033 >> v1:172.21.15.18:6789/0 conn(0x7f645c0c7cd0 legacy=0x7f645c1674a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:02.385 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.393+0000 7f6460ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:13:02.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.393+0000 7f6460ff9640 1 monclient: found mon.c 2024-03-29T11:13:02.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.393+0000 7f6460ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:02.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.393+0000 7f6460ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:13:02.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.393+0000 7f6460ff9640 1 -- 172.21.15.18:0/732766033 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f645c161070 con 0x7f645c0c51c0 2024-03-29T11:13:02.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.393+0000 7f6460ff9640 10 monclient: _finish_auth 0 2024-03-29T11:13:02.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.393+0000 7f6460ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:13:02.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.393+0000 7f6460ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:02.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.393+0000 7f6460ff9640 10 client.?.objecter ms_handle_connect 0x7f645c0c7cd0 2024-03-29T11:13:02.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.393+0000 7f6460ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:02.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.394+0000 7f6460ff9640 1 -- 172.21.15.18:0/732766033 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64580019d0 con 0x7f645c0c51c0 2024-03-29T11:13:02.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.394+0000 7f6460ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:02.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.394+0000 7f64627fc640 4 set_mon_vals no callback set 2024-03-29T11:13:02.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.394+0000 7f6460ff9640 1 -- 172.21.15.18:0/732766033 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f6458004ae0 con 0x7f645c0c51c0 2024-03-29T11:13:02.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.394+0000 7f6460ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:02.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.394+0000 7f6460ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:13:02.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.394+0000 7f6460ff9640 10 monclient: dump: 2024-03-29T11:13:02.387 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:13:02.387 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:02.387 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:02.387 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:02.387 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:02.387 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:02.387 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:13:02.387 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:13:02.387 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:13:02.387 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:02.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.395+0000 7f64695c8640 5 monclient: authenticate success, global_id 4400 2024-03-29T11:13:02.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.395+0000 7f64695c8640 10 monclient: _renew_subs 2024-03-29T11:13:02.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.395+0000 7f64695c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:13:02.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.395+0000 7f64695c8640 1 -- 172.21.15.18:0/732766033 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f645c0c1560 con 0x7f645c0c51c0 2024-03-29T11:13:02.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.395+0000 7f64695c8640 10 client.4400.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:13:02.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.395+0000 7f64695c8640 10 monclient: _renew_subs 2024-03-29T11:13:02.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.395+0000 7f64695c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:13:02.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.395+0000 7f64695c8640 1 -- 172.21.15.18:0/732766033 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f645c0c6750 con 0x7f645c0c51c0 2024-03-29T11:13:02.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.395+0000 7f64695c8640 1 librados: init done 2024-03-29T11:13:02.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.396+0000 7f64695c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:02.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.396+0000 7f64695c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:02.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.396+0000 7f64695c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:13:02.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.396+0000 7f64695c8640 1 -- 172.21.15.18:0/732766033 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f645c004cd0 con 0x7f645c0c51c0 2024-03-29T11:13:02.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.397+0000 7f6460ff9640 1 -- 172.21.15.18:0/732766033 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f645801aab0 con 0x7f645c0c51c0 2024-03-29T11:13:02.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.399+0000 7f6460ff9640 1 -- 172.21.15.18:0/732766033 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f6458084b30 con 0x7f645c0c51c0 2024-03-29T11:13:02.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.399+0000 7f6460ff9640 10 client.4400.objecter ms_dispatch 0x7f6464000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:13:02.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.399+0000 7f6460ff9640 3 client.4400.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:13:02.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.399+0000 7f6460ff9640 3 client.4400.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:13:02.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.399+0000 7f6460ff9640 20 client.4400.objecter dump_active .. 0 homeless 2024-03-29T11:13:02.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.403+0000 7f6463fff640 10 monclient: get_auth_request con 0x7f644806dbd0 auth_method 0 2024-03-29T11:13:02.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.403+0000 7f6460ff9640 10 client.4400.objecter ms_handle_connect 0x7f644806dbd0 2024-03-29T11:13:02.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.406+0000 7f6460ff9640 1 -- 172.21.15.18:0/732766033 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6458050240 con 0x7f645c0c51c0 2024-03-29T11:13:02.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.406+0000 7f6460ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:02.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.406+0000 7f6460ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:13:02.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.751+0000 7f64695c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:02.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.751+0000 7f64695c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:02.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.751+0000 7f64695c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:13:02.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.751+0000 7f64695c8640 1 -- 172.21.15.18:0/732766033 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f645c028e20 con 0x7f645c0c51c0 2024-03-29T11:13:02.730 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.752+0000 7f6460ff9640 1 -- 172.21.15.18:0/732766033 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f6458052340 con 0x7f645c0c51c0 2024-03-29T11:13:02.730 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.752+0000 7f6460ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:02.730 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.752+0000 7f6460ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:13:02.733 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:13:02.733 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:13:02.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.756+0000 7f64695c8640 10 librados: watch_flush enter 2024-03-29T11:13:02.734 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.756+0000 7f64695c8640 10 librados: watch_flush exit 2024-03-29T11:13:02.734 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.756+0000 7f64695c8640 20 client.4400.objecter shutdown clearing up homeless session... 2024-03-29T11:13:02.734 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.756+0000 7f64695c8640 10 client.4400.objecter successfully canceled tick 2024-03-29T11:13:02.734 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.756+0000 7f64695c8640 1 -- 172.21.15.18:0/732766033 >> v1:172.21.15.70:6816/3101871562 conn(0x7f644806dbd0 legacy=0x7f6448070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:02.734 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.756+0000 7f64695c8640 10 monclient: shutdown 2024-03-29T11:13:02.734 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.756+0000 7f64695c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:02.734 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.756+0000 7f64695c8640 1 -- 172.21.15.18:0/732766033 >> v1:172.21.15.18:6790/0 conn(0x7f645c0c51c0 legacy=0x7f645c160960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:02.735 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.757+0000 7f64695c8640 1 -- 172.21.15.18:0/732766033 shutdown_connections 2024-03-29T11:13:02.735 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.757+0000 7f64695c8640 1 -- 172.21.15.18:0/732766033 >> 172.21.15.18:0/732766033 conn(0x7f645c0058f0 msgr2=0x7f645c0043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:02.735 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.757+0000 7f64695c8640 1 -- 172.21.15.18:0/732766033 shutdown_connections 2024-03-29T11:13:02.735 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.757+0000 7f64695c8640 1 -- 172.21.15.18:0/732766033 wait complete. 2024-03-29T11:13:02.735 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:02.757+0000 7f64695c8640 1 librados: shutdown 2024-03-29T11:13:02.753 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:13:09.561 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:13:09.714 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.736+0000 7f9a8d9c8640 10 monclient: get_monmap_and_config 2024-03-29T11:13:09.714 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.736+0000 7f9a8d9c8640 10 monclient: build_initial_monmap 2024-03-29T11:13:09.715 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.736+0000 7f9a8d9c8640 10 monclient: monmap: 2024-03-29T11:13:09.715 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:13:09.715 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:09.715 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:13:09.737420+0000 2024-03-29T11:13:09.715 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:13:09.737420+0000 2024-03-29T11:13:09.715 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:09.715 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:09.715 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:13:09.715 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:13:09.715 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:13:09.715 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:09.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.737+0000 7f9a8d9c8640 1 Processor -- start 2024-03-29T11:13:09.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.737+0000 7f9a8d9c8640 1 -- start start 2024-03-29T11:13:09.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.737+0000 7f9a8d9c8640 10 monclient: init 2024-03-29T11:13:09.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.738+0000 7f9a8d9c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:09.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.738+0000 7f9a8d9c8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:13:09.717 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.738+0000 7f9a8d9c8640 10 monclient(hunting): picked mon.noname-a con 0x7f9a880b1f60 addr v1:172.21.15.18:6789/0 2024-03-29T11:13:09.717 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.738+0000 7f9a8d9c8640 10 monclient(hunting): picked mon.noname-c con 0x7f9a88071eb0 addr v1:172.21.15.70:6789/0 2024-03-29T11:13:09.717 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.738+0000 7f9a8d9c8640 10 monclient(hunting): picked mon.noname-b con 0x7f9a88143d10 addr v1:172.21.15.18:6790/0 2024-03-29T11:13:09.717 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.738+0000 7f9a8d9c8640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a8806ef80 con 0x7f9a880b1f60 2024-03-29T11:13:09.717 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.738+0000 7f9a8d9c8640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a8806f260 con 0x7f9a88071eb0 2024-03-29T11:13:09.717 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.738+0000 7f9a8d9c8640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a880b40c0 con 0x7f9a88143d10 2024-03-29T11:13:09.717 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.738+0000 7f9a8d9c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:09.717 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.738+0000 7f9a8d9c8640 10 monclient(hunting): authenticate will time out at 1016.010494s 2024-03-29T11:13:09.719 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.741+0000 7f9a877fe640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f9a88143d10 0x7f9a88146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:58874/0 (socket says 172.21.15.18:58874) 2024-03-29T11:13:09.719 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.741+0000 7f9a877fe640 1 -- 172.21.15.18:0/634427848 learned_addr learned my addr 172.21.15.18:0/634427848 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:13:09.724 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.743+0000 7f9a85ffb640 1 -- 172.21.15.18:0/634427848 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1316025240 0 0) 0x7f9a8806ef80 con 0x7f9a880b1f60 2024-03-29T11:13:09.724 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.743+0000 7f9a85ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:13:09.724 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.743+0000 7f9a85ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:09.724 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.743+0000 7f9a85ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:09.724 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.743+0000 7f9a85ffb640 10 monclient(hunting): my global_id is 4353 2024-03-29T11:13:09.724 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.743+0000 7f9a85ffb640 1 -- 172.21.15.18:0/634427848 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a74003520 con 0x7f9a880b1f60 2024-03-29T11:13:09.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.743+0000 7f9a85ffb640 1 -- 172.21.15.18:0/634427848 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2169199728 0 0) 0x7f9a74003520 con 0x7f9a880b1f60 2024-03-29T11:13:09.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.743+0000 7f9a85ffb640 1 -- 172.21.15.18:0/634427848 >> v1:172.21.15.18:6790/0 conn(0x7f9a88143d10 legacy=0x7f9a88146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:09.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.743+0000 7f9a85ffb640 1 -- 172.21.15.18:0/634427848 >> v1:172.21.15.70:6789/0 conn(0x7f9a88071eb0 legacy=0x7f9a88146470 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:13:09.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.743+0000 7f9a85ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:13:09.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.743+0000 7f9a85ffb640 1 monclient: found mon.noname-a 2024-03-29T11:13:09.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.743+0000 7f9a85ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:09.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.743+0000 7f9a85ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-29T11:13:09.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.743+0000 7f9a85ffb640 1 -- 172.21.15.18:0/634427848 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a88155d70 con 0x7f9a880b1f60 2024-03-29T11:13:09.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.743+0000 7f9a85ffb640 10 monclient: _finish_auth 0 2024-03-29T11:13:09.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.743+0000 7f9a85ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:13:09.725 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.743+0000 7f9a85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:09.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.743+0000 7f9a85ffb640 1 -- 172.21.15.18:0/634427848 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f9a80001690 con 0x7f9a880b1f60 2024-03-29T11:13:09.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.743+0000 7f9a85ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:09.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.743+0000 7f9a85ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:13:09.726 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.743+0000 7f9a85ffb640 10 monclient: dump: 2024-03-29T11:13:09.726 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:13:09.726 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:09.726 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:09.726 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:09.726 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:09.726 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:09.727 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:13:09.727 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:13:09.727 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:13:09.727 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:09.727 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.743+0000 7f9a85ffb640 1 -- 172.21.15.18:0/634427848 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a80001860 con 0x7f9a880b1f60 2024-03-29T11:13:09.727 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.743+0000 7f9a85ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:09.727 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.743+0000 7f9a85ffb640 1 -- 172.21.15.18:0/634427848 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f9a80001cb0 con 0x7f9a880b1f60 2024-03-29T11:13:09.727 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.744+0000 7f9a85ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:09.727 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.744+0000 7f9a85ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:13:09.727 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.744+0000 7f9a85ffb640 10 monclient: dump: 2024-03-29T11:13:09.727 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:13:09.728 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:09.728 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:09.728 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:09.728 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:09.728 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:09.729 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:13:09.729 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:13:09.729 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:13:09.729 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:09.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.744+0000 7f9a8d9c8640 5 monclient: authenticate success, global_id 4353 2024-03-29T11:13:09.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.744+0000 7f9a8d9c8640 10 monclient: get_monmap_and_config success 2024-03-29T11:13:09.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.744+0000 7f9a8d9c8640 4 set_mon_vals no callback set 2024-03-29T11:13:09.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.744+0000 7f9a8d9c8640 10 monclient: shutdown 2024-03-29T11:13:09.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.744+0000 7f9a8d9c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:09.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.744+0000 7f9a8d9c8640 1 -- 172.21.15.18:0/634427848 >> v1:172.21.15.18:6789/0 conn(0x7f9a880b1f60 legacy=0x7f9a88143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:09.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.745+0000 7f9a8d9c8640 1 -- 172.21.15.18:0/634427848 shutdown_connections 2024-03-29T11:13:09.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.745+0000 7f9a8d9c8640 1 -- 172.21.15.18:0/634427848 >> 172.21.15.18:0/634427848 conn(0x7f9a88066a00 msgr2=0x7f9a88066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:09.729 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.746+0000 7f9a8d9c8640 1 -- 172.21.15.18:0/634427848 shutdown_connections 2024-03-29T11:13:09.730 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.746+0000 7f9a8d9c8640 1 -- 172.21.15.18:0/634427848 wait complete. 2024-03-29T11:13:09.734 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.747+0000 7f9a8d9c8640 10 monclient: build_initial_monmap 2024-03-29T11:13:09.734 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.747+0000 7f9a8d9c8640 10 monclient: monmap: 2024-03-29T11:13:09.734 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:13:09.734 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:13:09.734 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:13:09.734 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:13:09.734 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:09.734 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:09.734 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:13:09.734 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:13:09.734 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:13:09.735 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:09.735 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.747+0000 7f9a8d9c8640 1 librados: starting msgr at 2024-03-29T11:13:09.735 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.747+0000 7f9a8d9c8640 1 librados: starting objecter 2024-03-29T11:13:09.735 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.747+0000 7f9a8d9c8640 1 Processor -- start 2024-03-29T11:13:09.735 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.747+0000 7f9a8d9c8640 1 -- start start 2024-03-29T11:13:09.735 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.747+0000 7f9a8d9c8640 1 librados: setting wanted keys 2024-03-29T11:13:09.735 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.747+0000 7f9a8d9c8640 1 librados: calling monclient init 2024-03-29T11:13:09.735 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.747+0000 7f9a8d9c8640 10 monclient: init 2024-03-29T11:13:09.735 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.748+0000 7f9a8d9c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:09.735 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.748+0000 7f9a8d9c8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:13:09.735 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.748+0000 7f9a8d9c8640 10 monclient(hunting): picked mon.noname-b con 0x7f9a880b1f60 addr v1:172.21.15.70:6789/0 2024-03-29T11:13:09.735 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.748+0000 7f9a8d9c8640 10 monclient(hunting): picked mon.noname-c con 0x7f9a88143d10 addr v1:172.21.15.18:6790/0 2024-03-29T11:13:09.735 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.748+0000 7f9a8d9c8640 10 monclient(hunting): picked mon.noname-a con 0x7f9a881df440 addr v1:172.21.15.18:6789/0 2024-03-29T11:13:09.735 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.748+0000 7f9a8d9c8640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a8813bc00 con 0x7f9a881df440 2024-03-29T11:13:09.736 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.748+0000 7f9a8d9c8640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a8814b3e0 con 0x7f9a880b1f60 2024-03-29T11:13:09.736 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.748+0000 7f9a8d9c8640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a88149e70 con 0x7f9a88143d10 2024-03-29T11:13:09.736 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.748+0000 7f9a8d9c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:09.736 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.748+0000 7f9a8d9c8640 10 monclient(hunting): authenticate will time out at 1016.020567s 2024-03-29T11:13:09.736 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.750+0000 7f9a867fc640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f9a88143d10 0x7f9a8813ade0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:59016/0 (socket says 172.21.15.18:59016) 2024-03-29T11:13:09.736 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.750+0000 7f9a867fc640 1 -- 172.21.15.18:0/2315423807 learned_addr learned my addr 172.21.15.18:0/2315423807 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:13:09.736 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.750+0000 7f9a7ffff640 10 client.?.objecter ms_handle_connect 0x7f9a88143d10 2024-03-29T11:13:09.736 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.750+0000 7f9a7ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:09.736 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.751+0000 7f9a7ffff640 10 client.?.objecter ms_handle_connect 0x7f9a881df440 2024-03-29T11:13:09.736 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.751+0000 7f9a7ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:09.736 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.751+0000 7f9a7ffff640 1 -- 172.21.15.18:0/2315423807 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3829224830 0 0) 0x7f9a88149e70 con 0x7f9a88143d10 2024-03-29T11:13:09.736 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.751+0000 7f9a7ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:13:09.737 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.751+0000 7f9a7ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:09.737 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.751+0000 7f9a7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:09.737 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.751+0000 7f9a7ffff640 10 monclient(hunting): my global_id is 4406 2024-03-29T11:13:09.737 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.752+0000 7f9a7ffff640 1 -- 172.21.15.18:0/2315423807 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a74003160 con 0x7f9a88143d10 2024-03-29T11:13:09.737 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.752+0000 7f9a7ffff640 1 -- 172.21.15.18:0/2315423807 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4057764878 0 0) 0x7f9a8813bc00 con 0x7f9a881df440 2024-03-29T11:13:09.737 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.752+0000 7f9a7ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:13:09.737 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.752+0000 7f9a7ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:09.737 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.752+0000 7f9a7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:09.737 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.752+0000 7f9a7ffff640 10 monclient(hunting): my global_id is 4356 2024-03-29T11:13:09.737 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.752+0000 7f9a7ffff640 1 -- 172.21.15.18:0/2315423807 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a88149e70 con 0x7f9a881df440 2024-03-29T11:13:09.738 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.752+0000 7f9a7ffff640 1 -- 172.21.15.18:0/2315423807 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1498434794 0 0) 0x7f9a88149e70 con 0x7f9a881df440 2024-03-29T11:13:09.738 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.752+0000 7f9a7ffff640 1 -- 172.21.15.18:0/2315423807 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a8813bc00 con 0x7f9a881df440 2024-03-29T11:13:09.738 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.752+0000 7f9a7ffff640 1 -- 172.21.15.18:0/2315423807 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f9a78001b90 con 0x7f9a881df440 2024-03-29T11:13:09.738 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.753+0000 7f9a7ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:09.738 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.753+0000 7f9a7ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:13:09.738 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.753+0000 7f9a7ffff640 10 monclient(hunting): dump: 2024-03-29T11:13:09.738 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:13:09.738 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:09.738 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:09.739 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:09.739 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:09.739 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:09.739 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:13:09.739 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:13:09.739 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:13:09.739 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:09.739 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.753+0000 7f9a7ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:13:09.739 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.753+0000 7f9a7ffff640 1 -- 172.21.15.18:0/2315423807 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3670111636 0 0) 0x7f9a74003160 con 0x7f9a88143d10 2024-03-29T11:13:09.739 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.753+0000 7f9a7ffff640 1 -- 172.21.15.18:0/2315423807 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a88149e70 con 0x7f9a88143d10 2024-03-29T11:13:09.739 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.753+0000 7f9a7ffff640 1 -- 172.21.15.18:0/2315423807 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f9a70001870 con 0x7f9a88143d10 2024-03-29T11:13:09.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.753+0000 7f9a7ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:09.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.753+0000 7f9a7ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:13:09.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.753+0000 7f9a7ffff640 10 monclient(hunting): dump: 2024-03-29T11:13:09.740 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:13:09.740 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:09.740 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:09.740 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:09.740 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:09.740 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:09.740 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:13:09.740 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:13:09.740 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:13:09.741 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:09.741 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.753+0000 7f9a7ffff640 1 -- 172.21.15.18:0/2315423807 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4118986932 0 0) 0x7f9a8813bc00 con 0x7f9a881df440 2024-03-29T11:13:09.741 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.753+0000 7f9a7ffff640 1 -- 172.21.15.18:0/2315423807 >> v1:172.21.15.18:6790/0 conn(0x7f9a88143d10 legacy=0x7f9a8813ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:09.741 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.753+0000 7f9a7ffff640 1 -- 172.21.15.18:0/2315423807 >> v1:172.21.15.70:6789/0 conn(0x7f9a880b1f60 legacy=0x7f9a881ded30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:13:09.741 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.754+0000 7f9a7ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:13:09.741 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.754+0000 7f9a7ffff640 1 monclient: found mon.a 2024-03-29T11:13:09.741 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.754+0000 7f9a7ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:09.741 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.754+0000 7f9a7ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:13:09.741 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.754+0000 7f9a7ffff640 1 -- 172.21.15.18:0/2315423807 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a88149b00 con 0x7f9a881df440 2024-03-29T11:13:09.742 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.754+0000 7f9a7ffff640 10 monclient: _finish_auth 0 2024-03-29T11:13:09.742 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.754+0000 7f9a7ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:13:09.742 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.754+0000 7f9a7ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:09.742 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.754+0000 7f9a8d9c8640 5 monclient: authenticate success, global_id 4356 2024-03-29T11:13:09.742 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.754+0000 7f9a8d9c8640 10 monclient: _renew_subs 2024-03-29T11:13:09.742 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.754+0000 7f9a8d9c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:13:09.742 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.754+0000 7f9a8d9c8640 1 -- 172.21.15.18:0/2315423807 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a8814a390 con 0x7f9a881df440 2024-03-29T11:13:09.742 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.754+0000 7f9a8d9c8640 10 client.4356.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:13:09.742 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.754+0000 7f9a8d9c8640 10 monclient: _renew_subs 2024-03-29T11:13:09.742 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.754+0000 7f9a8d9c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:13:09.742 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.754+0000 7f9a8d9c8640 1 -- 172.21.15.18:0/2315423807 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a88154dc0 con 0x7f9a881df440 2024-03-29T11:13:09.742 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.754+0000 7f9a8d9c8640 1 librados: init done 2024-03-29T11:13:09.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.756+0000 7f9a8d9c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:09.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.756+0000 7f9a8d9c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:09.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.756+0000 7f9a8d9c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:13:09.743 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.756+0000 7f9a8d9c8640 1 -- 172.21.15.18:0/2315423807 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a8814a000 con 0x7f9a881df440 2024-03-29T11:13:09.745 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.757+0000 7f9a7ffff640 1 -- 172.21.15.18:0/2315423807 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a78001fa0 con 0x7f9a881df440 2024-03-29T11:13:09.745 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.757+0000 7f9a7ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:09.745 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.757+0000 7f9a857fa640 4 set_mon_vals no callback set 2024-03-29T11:13:09.745 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.757+0000 7f9a7ffff640 1 -- 172.21.15.18:0/2315423807 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f9a78004d00 con 0x7f9a881df440 2024-03-29T11:13:09.745 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.757+0000 7f9a7ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:09.745 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.757+0000 7f9a7ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:13:09.746 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.757+0000 7f9a7ffff640 10 monclient: dump: 2024-03-29T11:13:09.746 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:13:09.746 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:09.746 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:09.746 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:09.746 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:09.746 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:09.746 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:13:09.746 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:13:09.746 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:13:09.746 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:09.746 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.760+0000 7f9a7ffff640 1 -- 172.21.15.18:0/2315423807 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f9a78004e60 con 0x7f9a881df440 2024-03-29T11:13:09.747 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.761+0000 7f9a7ffff640 1 -- 172.21.15.18:0/2315423807 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f9a78084c30 con 0x7f9a881df440 2024-03-29T11:13:09.747 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.761+0000 7f9a7ffff640 10 client.4356.objecter ms_dispatch 0x7f9a88000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:13:09.747 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.761+0000 7f9a7ffff640 3 client.4356.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:13:09.747 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.761+0000 7f9a7ffff640 3 client.4356.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:13:09.747 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.761+0000 7f9a7ffff640 20 client.4356.objecter dump_active .. 0 homeless 2024-03-29T11:13:09.747 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.762+0000 7f9a86ffd640 10 monclient: get_auth_request con 0x7f9a7406e050 auth_method 0 2024-03-29T11:13:09.747 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.762+0000 7f9a7ffff640 10 client.4356.objecter ms_handle_connect 0x7f9a7406e050 2024-03-29T11:13:09.747 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.767+0000 7f9a7ffff640 1 -- 172.21.15.18:0/2315423807 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9a78050510 con 0x7f9a881df440 2024-03-29T11:13:09.747 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.767+0000 7f9a7ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:09.747 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:09.767+0000 7f9a7ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:13:10.086 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:10.108+0000 7f9a8d9c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:10.086 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:10.108+0000 7f9a8d9c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:10.086 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:10.108+0000 7f9a8d9c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:13:10.086 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:10.108+0000 7f9a8d9c8640 1 -- 172.21.15.18:0/2315423807 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a88075f10 con 0x7f9a881df440 2024-03-29T11:13:10.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:10.109+0000 7f9a7ffff640 1 -- 172.21.15.18:0/2315423807 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f9a78052440 con 0x7f9a881df440 2024-03-29T11:13:10.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:10.109+0000 7f9a7ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:10.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:10.109+0000 7f9a7ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:13:10.088 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:13:10.088 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:13:10.091 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:10.113+0000 7f9a8d9c8640 10 librados: watch_flush enter 2024-03-29T11:13:10.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:10.114+0000 7f9a8d9c8640 10 librados: watch_flush exit 2024-03-29T11:13:10.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:10.114+0000 7f9a8d9c8640 20 client.4356.objecter shutdown clearing up homeless session... 2024-03-29T11:13:10.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:10.114+0000 7f9a8d9c8640 10 client.4356.objecter successfully canceled tick 2024-03-29T11:13:10.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:10.115+0000 7f9a8d9c8640 1 -- 172.21.15.18:0/2315423807 >> v1:172.21.15.70:6816/3101871562 conn(0x7f9a7406e050 legacy=0x7f9a74070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:10.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:10.115+0000 7f9a8d9c8640 10 monclient: shutdown 2024-03-29T11:13:10.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:10.115+0000 7f9a8d9c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:10.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:10.115+0000 7f9a8d9c8640 1 -- 172.21.15.18:0/2315423807 >> v1:172.21.15.18:6789/0 conn(0x7f9a881df440 legacy=0x7f9a8813b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:10.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:10.115+0000 7f9a8d9c8640 1 -- 172.21.15.18:0/2315423807 shutdown_connections 2024-03-29T11:13:10.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:10.115+0000 7f9a8d9c8640 1 -- 172.21.15.18:0/2315423807 >> 172.21.15.18:0/2315423807 conn(0x7f9a88066a00 msgr2=0x7f9a880b7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:10.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:10.115+0000 7f9a8d9c8640 1 -- 172.21.15.18:0/2315423807 shutdown_connections 2024-03-29T11:13:10.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:10.115+0000 7f9a8d9c8640 1 -- 172.21.15.18:0/2315423807 wait complete. 2024-03-29T11:13:10.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:10.115+0000 7f9a8d9c8640 1 librados: shutdown 2024-03-29T11:13:10.113 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:13:16.922 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:13:17.066 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.088+0000 7fafc6a20640 10 monclient: get_monmap_and_config 2024-03-29T11:13:17.066 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.088+0000 7fafc6a20640 10 monclient: build_initial_monmap 2024-03-29T11:13:17.066 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.088+0000 7fafc6a20640 10 monclient: monmap: 2024-03-29T11:13:17.066 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:13:17.066 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:17.066 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:13:17.088954+0000 2024-03-29T11:13:17.066 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:13:17.088954+0000 2024-03-29T11:13:17.066 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:17.066 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:17.066 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:13:17.066 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:13:17.067 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:13:17.067 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:17.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.090+0000 7fafc6a20640 1 Processor -- start 2024-03-29T11:13:17.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.090+0000 7fafc6a20640 1 -- start start 2024-03-29T11:13:17.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.090+0000 7fafc6a20640 10 monclient: init 2024-03-29T11:13:17.070 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.091+0000 7fafc6a20640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:17.070 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.091+0000 7fafc6a20640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:13:17.071 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.091+0000 7fafc6a20640 10 monclient(hunting): picked mon.noname-a con 0x7fafc01457f0 addr v1:172.21.15.18:6789/0 2024-03-29T11:13:17.071 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.091+0000 7fafc6a20640 10 monclient(hunting): picked mon.noname-b con 0x7fafc013f650 addr v1:172.21.15.18:6790/0 2024-03-29T11:13:17.071 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.091+0000 7fafc6a20640 10 monclient(hunting): picked mon.noname-c con 0x7fafc00a9d70 addr v1:172.21.15.70:6789/0 2024-03-29T11:13:17.071 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.091+0000 7fafc6a20640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafc00aaef0 con 0x7fafc01457f0 2024-03-29T11:13:17.071 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.091+0000 7fafc6a20640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafc0145bd0 con 0x7fafc00a9d70 2024-03-29T11:13:17.071 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.091+0000 7fafc6a20640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafc0145eb0 con 0x7fafc013f650 2024-03-29T11:13:17.071 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.091+0000 7fafc6a20640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:17.071 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.091+0000 7fafc6a20640 10 monclient(hunting): authenticate will time out at 1023.363408s 2024-03-29T11:13:17.071 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.094+0000 7fafbf7fe640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fafc013f650 0x7fafc00b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:51012/0 (socket says 172.21.15.18:51012) 2024-03-29T11:13:17.072 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.094+0000 7fafbf7fe640 1 -- 172.21.15.18:0/437718499 learned_addr learned my addr 172.21.15.18:0/437718499 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:13:17.075 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.095+0000 7fafbeffd640 1 -- 172.21.15.18:0/437718499 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2758875029 0 0) 0x7fafc00aaef0 con 0x7fafc01457f0 2024-03-29T11:13:17.076 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.095+0000 7fafbeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:13:17.076 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.095+0000 7fafbeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:17.076 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.095+0000 7fafbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:17.076 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.095+0000 7fafbeffd640 10 monclient(hunting): my global_id is 4359 2024-03-29T11:13:17.076 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.095+0000 7fafbeffd640 1 -- 172.21.15.18:0/437718499 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafa8003520 con 0x7fafc01457f0 2024-03-29T11:13:17.076 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.095+0000 7fafbeffd640 1 -- 172.21.15.18:0/437718499 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 406394558 0 0) 0x7fafa8003520 con 0x7fafc01457f0 2024-03-29T11:13:17.076 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.095+0000 7fafbeffd640 1 -- 172.21.15.18:0/437718499 >> v1:172.21.15.18:6790/0 conn(0x7fafc013f650 legacy=0x7fafc00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:13:17.076 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.095+0000 7fafbeffd640 1 -- 172.21.15.18:0/437718499 >> v1:172.21.15.70:6789/0 conn(0x7fafc00a9d70 legacy=0x7fafc014d8c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:13:17.076 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.095+0000 7fafbeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:13:17.076 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.095+0000 7fafbeffd640 1 monclient: found mon.noname-a 2024-03-29T11:13:17.076 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.095+0000 7fafbeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:17.077 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.095+0000 7fafbeffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-29T11:13:17.077 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.095+0000 7fafbeffd640 1 -- 172.21.15.18:0/437718499 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc0146340 con 0x7fafc01457f0 2024-03-29T11:13:17.077 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.095+0000 7fafbeffd640 10 monclient: _finish_auth 0 2024-03-29T11:13:17.077 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.095+0000 7fafbeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:13:17.077 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.095+0000 7fafbeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:17.077 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.095+0000 7fafbeffd640 1 -- 172.21.15.18:0/437718499 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fafb8001a10 con 0x7fafc01457f0 2024-03-29T11:13:17.077 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.095+0000 7fafbeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:17.077 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.095+0000 7fafbeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:13:17.077 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.095+0000 7fafbeffd640 10 monclient: dump: 2024-03-29T11:13:17.077 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:13:17.077 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:17.077 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:17.077 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:17.077 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:17.077 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:17.078 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:13:17.078 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:13:17.078 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:13:17.078 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:17.078 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.096+0000 7fafbeffd640 1 -- 172.21.15.18:0/437718499 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fafb8001be0 con 0x7fafc01457f0 2024-03-29T11:13:17.078 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.096+0000 7fafbeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:17.078 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.096+0000 7fafbeffd640 1 -- 172.21.15.18:0/437718499 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fafb8002030 con 0x7fafc01457f0 2024-03-29T11:13:17.078 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.096+0000 7fafbeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:17.078 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.096+0000 7fafbeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:13:17.078 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.096+0000 7fafbeffd640 10 monclient: dump: 2024-03-29T11:13:17.078 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:13:17.078 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:17.078 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:17.078 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:17.079 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:17.079 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:17.079 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:13:17.079 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:13:17.079 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:13:17.079 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:17.079 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.096+0000 7fafc6a20640 5 monclient: authenticate success, global_id 4359 2024-03-29T11:13:17.079 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.096+0000 7fafc6a20640 10 monclient: get_monmap_and_config success 2024-03-29T11:13:17.079 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.096+0000 7fafc6a20640 4 set_mon_vals no callback set 2024-03-29T11:13:17.080 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.097+0000 7fafc6a20640 10 monclient: shutdown 2024-03-29T11:13:17.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.098+0000 7fafc6a20640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:17.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.098+0000 7fafc6a20640 1 -- 172.21.15.18:0/437718499 >> v1:172.21.15.18:6789/0 conn(0x7fafc01457f0 legacy=0x7fafc013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:17.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.098+0000 7fafc6a20640 1 -- 172.21.15.18:0/437718499 shutdown_connections 2024-03-29T11:13:17.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.098+0000 7fafc6a20640 1 -- 172.21.15.18:0/437718499 >> 172.21.15.18:0/437718499 conn(0x7fafc0066a00 msgr2=0x7fafc0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:17.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.100+0000 7fafc6a20640 1 -- 172.21.15.18:0/437718499 shutdown_connections 2024-03-29T11:13:17.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.100+0000 7fafc6a20640 1 -- 172.21.15.18:0/437718499 wait complete. 2024-03-29T11:13:17.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.102+0000 7fafc6a20640 10 monclient: build_initial_monmap 2024-03-29T11:13:17.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.102+0000 7fafc6a20640 10 monclient: monmap: 2024-03-29T11:13:17.081 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:13:17.081 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:13:17.081 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:13:17.081 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:13:17.081 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:17.082 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:17.082 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:13:17.082 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:13:17.082 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:13:17.082 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:17.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.102+0000 7fafc6a20640 1 librados: starting msgr at 2024-03-29T11:13:17.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.102+0000 7fafc6a20640 1 librados: starting objecter 2024-03-29T11:13:17.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.103+0000 7fafc6a20640 1 Processor -- start 2024-03-29T11:13:17.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.103+0000 7fafc6a20640 1 -- start start 2024-03-29T11:13:17.083 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.103+0000 7fafc6a20640 1 librados: setting wanted keys 2024-03-29T11:13:17.083 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.103+0000 7fafc6a20640 1 librados: calling monclient init 2024-03-29T11:13:17.083 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.103+0000 7fafc6a20640 10 monclient: init 2024-03-29T11:13:17.084 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.104+0000 7fafc6a20640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:17.084 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.104+0000 7fafc6a20640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:13:17.084 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.104+0000 7fafc6a20640 10 monclient(hunting): picked mon.noname-a con 0x7fafc00a9d70 addr v1:172.21.15.18:6789/0 2024-03-29T11:13:17.086 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.107+0000 7fafbffff640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fafc00a9d70 0x7fafc013a270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:47627/0 (socket says 172.21.15.18:47627) 2024-03-29T11:13:17.086 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.107+0000 7fafbffff640 1 -- 172.21.15.18:0/882394337 learned_addr learned my addr 172.21.15.18:0/882394337 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:13:17.086 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.107+0000 7fafbcff9640 10 client.?.objecter ms_handle_connect 0x7fafc00a9d70 2024-03-29T11:13:17.086 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.107+0000 7fafbcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:17.086 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.108+0000 7fafc6a20640 10 monclient(hunting): picked mon.noname-b con 0x7fafc013f650 addr v1:172.21.15.70:6789/0 2024-03-29T11:13:17.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.108+0000 7fafc6a20640 10 monclient(hunting): picked mon.noname-c con 0x7fafc01457f0 addr v1:172.21.15.18:6790/0 2024-03-29T11:13:17.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.108+0000 7fafc6a20640 1 -- 172.21.15.18:0/882394337 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafc0139900 con 0x7fafc00a9d70 2024-03-29T11:13:17.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.108+0000 7fafc6a20640 1 -- 172.21.15.18:0/882394337 --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafc013a980 con 0x7fafc013f650 2024-03-29T11:13:17.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.108+0000 7fafc6a20640 1 -- 172.21.15.18:0/882394337 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafc01dbbd0 con 0x7fafc01457f0 2024-03-29T11:13:17.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.108+0000 7fafc6a20640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:17.088 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.108+0000 7fafc6a20640 10 monclient(hunting): authenticate will time out at 1023.380846s 2024-03-29T11:13:17.088 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.109+0000 7fafbcff9640 10 client.?.objecter ms_handle_connect 0x7fafc01457f0 2024-03-29T11:13:17.089 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.109+0000 7fafbcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:17.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.112+0000 7fafbcff9640 1 -- 172.21.15.18:0/882394337 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3115695088 0 0) 0x7fafc0139900 con 0x7fafc00a9d70 2024-03-29T11:13:17.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.112+0000 7fafbcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:13:17.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.112+0000 7fafbcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:17.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.112+0000 7fafbcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:17.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.112+0000 7fafbcff9640 10 monclient(hunting): my global_id is 4362 2024-03-29T11:13:17.092 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.112+0000 7fafbcff9640 1 -- 172.21.15.18:0/882394337 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafac004eb0 con 0x7fafc00a9d70 2024-03-29T11:13:17.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.112+0000 7fafbcff9640 1 -- 172.21.15.18:0/882394337 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3191418237 0 0) 0x7fafc01dbbd0 con 0x7fafc01457f0 2024-03-29T11:13:17.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.112+0000 7fafbcff9640 10 monclient(hunting): _negotiate 2024-03-29T11:13:17.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.112+0000 7fafbcff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:17.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.112+0000 7fafbcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:17.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.112+0000 7fafbcff9640 10 monclient(hunting): my global_id is 4409 2024-03-29T11:13:17.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.112+0000 7fafbcff9640 1 -- 172.21.15.18:0/882394337 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafc0139900 con 0x7fafc01457f0 2024-03-29T11:13:17.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.113+0000 7fafbcff9640 10 client.?.objecter ms_handle_connect 0x7fafc013f650 2024-03-29T11:13:17.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.113+0000 7fafbcff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:17.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.113+0000 7fafbcff9640 1 -- 172.21.15.18:0/882394337 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2431700720 0 0) 0x7fafc0139900 con 0x7fafc01457f0 2024-03-29T11:13:17.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.113+0000 7fafbcff9640 1 -- 172.21.15.18:0/882394337 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fafc01dbbd0 con 0x7fafc01457f0 2024-03-29T11:13:17.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.113+0000 7fafbcff9640 1 -- 172.21.15.18:0/882394337 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fafb4001730 con 0x7fafc01457f0 2024-03-29T11:13:17.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.113+0000 7fafbcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:17.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.113+0000 7fafbcff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:13:17.093 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.113+0000 7fafbcff9640 10 monclient(hunting): dump: 2024-03-29T11:13:17.094 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:13:17.094 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:17.094 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:17.094 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:17.094 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:17.094 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:17.094 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:13:17.094 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:13:17.094 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:13:17.094 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:17.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.113+0000 7fafbcff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:13:17.094 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.113+0000 7fafbcff9640 1 -- 172.21.15.18:0/882394337 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3428319860 0 0) 0x7fafac004eb0 con 0x7fafc00a9d70 2024-03-29T11:13:17.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.113+0000 7fafbcff9640 1 -- 172.21.15.18:0/882394337 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fafc0139900 con 0x7fafc00a9d70 2024-03-29T11:13:17.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.113+0000 7fafbcff9640 1 -- 172.21.15.18:0/882394337 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fafb8001fb0 con 0x7fafc00a9d70 2024-03-29T11:13:17.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.113+0000 7fafbcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:17.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.113+0000 7fafbcff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:13:17.097 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.113+0000 7fafbcff9640 10 monclient(hunting): dump: 2024-03-29T11:13:17.097 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:13:17.097 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:17.097 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:17.097 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:17.097 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:17.098 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:17.098 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:13:17.098 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:13:17.098 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:13:17.098 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:17.098 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.113+0000 7fafbcff9640 1 -- 172.21.15.18:0/882394337 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1591356232 0 0) 0x7fafc01dbbd0 con 0x7fafc01457f0 2024-03-29T11:13:17.098 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.113+0000 7fafbcff9640 1 -- 172.21.15.18:0/882394337 >> v1:172.21.15.70:6789/0 conn(0x7fafc013f650 legacy=0x7fafc01dadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:17.098 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.113+0000 7fafbcff9640 1 -- 172.21.15.18:0/882394337 >> v1:172.21.15.18:6789/0 conn(0x7fafc00a9d70 legacy=0x7fafc013a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:17.098 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.113+0000 7fafbcff9640 10 monclient: _finish_hunting 0 2024-03-29T11:13:17.098 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.113+0000 7fafbcff9640 1 monclient: found mon.c 2024-03-29T11:13:17.098 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.113+0000 7fafbcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:17.098 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.113+0000 7fafbcff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:13:17.098 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.113+0000 7fafbcff9640 1 -- 172.21.15.18:0/882394337 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc0139b70 con 0x7fafc01457f0 2024-03-29T11:13:17.098 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.113+0000 7fafbcff9640 10 monclient: _finish_auth 0 2024-03-29T11:13:17.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.113+0000 7fafbcff9640 10 monclient: _check_auth_tickets 2024-03-29T11:13:17.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.113+0000 7fafbcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:17.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.113+0000 7fafc6a20640 5 monclient: authenticate success, global_id 4409 2024-03-29T11:13:17.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.113+0000 7fafc6a20640 10 monclient: _renew_subs 2024-03-29T11:13:17.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.113+0000 7fafc6a20640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:13:17.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.113+0000 7fafc6a20640 1 -- 172.21.15.18:0/882394337 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafc0139e50 con 0x7fafc01457f0 2024-03-29T11:13:17.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.113+0000 7fafc6a20640 10 client.4409.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:13:17.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.113+0000 7fafc6a20640 10 monclient: _renew_subs 2024-03-29T11:13:17.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.113+0000 7fafc6a20640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:13:17.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.113+0000 7fafc6a20640 1 -- 172.21.15.18:0/882394337 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fafc0138bc0 con 0x7fafc01457f0 2024-03-29T11:13:17.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.113+0000 7fafc6a20640 1 librados: init done 2024-03-29T11:13:17.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.117+0000 7fafbcff9640 1 -- 172.21.15.18:0/882394337 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fafb4001d20 con 0x7fafc01457f0 2024-03-29T11:13:17.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.117+0000 7fafbcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:17.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.117+0000 7fafbcff9640 1 -- 172.21.15.18:0/882394337 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fafb4003830 con 0x7fafc01457f0 2024-03-29T11:13:17.099 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.117+0000 7fafbcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:17.100 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.117+0000 7fafbcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:13:17.100 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.117+0000 7fafbcff9640 10 monclient: dump: 2024-03-29T11:13:17.100 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:13:17.100 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:17.100 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:17.100 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:17.100 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:17.100 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:17.100 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:13:17.100 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:13:17.100 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:13:17.100 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:17.100 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.117+0000 7fafbcff9640 1 -- 172.21.15.18:0/882394337 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7fafb4019800 con 0x7fafc01457f0 2024-03-29T11:13:17.100 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.118+0000 7fafbcff9640 1 -- 172.21.15.18:0/882394337 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7fafb4083800 con 0x7fafc01457f0 2024-03-29T11:13:17.101 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.118+0000 7fafbcff9640 10 client.4409.objecter ms_dispatch 0x7fafc0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:13:17.101 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.118+0000 7fafbcff9640 3 client.4409.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:13:17.101 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.118+0000 7fafbcff9640 3 client.4409.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:13:17.101 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.118+0000 7fafbcff9640 20 client.4409.objecter dump_active .. 0 homeless 2024-03-29T11:13:17.101 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.118+0000 7fafbdffb640 4 set_mon_vals no callback set 2024-03-29T11:13:17.101 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.119+0000 7fafbffff640 10 monclient: get_auth_request con 0x7fafac0768f0 auth_method 0 2024-03-29T11:13:17.101 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.119+0000 7fafa67fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:17.101 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.119+0000 7fafa67fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:17.102 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.119+0000 7fafa67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:13:17.102 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.119+0000 7fafa67fc640 1 -- 172.21.15.18:0/882394337 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafa0005080 con 0x7fafc01457f0 2024-03-29T11:13:17.102 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.121+0000 7fafbcff9640 10 client.4409.objecter ms_handle_connect 0x7fafac0768f0 2024-03-29T11:13:17.106 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.128+0000 7fafbcff9640 1 -- 172.21.15.18:0/882394337 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fafb404f160 con 0x7fafc01457f0 2024-03-29T11:13:17.106 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.128+0000 7fafbcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:17.106 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.128+0000 7fafbcff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:13:17.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.457+0000 7fafa67fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:17.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.457+0000 7fafa67fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:17.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.457+0000 7fafa67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:13:17.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.457+0000 7fafa67fc640 1 -- 172.21.15.18:0/882394337 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fafa0005330 con 0x7fafc01457f0 2024-03-29T11:13:17.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.459+0000 7fafbcff9640 1 -- 172.21.15.18:0/882394337 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7fafb4051090 con 0x7fafc01457f0 2024-03-29T11:13:17.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.459+0000 7fafbcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:17.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.459+0000 7fafbcff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:13:17.437 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:13:17.437 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:13:17.444 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.463+0000 7fafa67fc640 10 librados: watch_flush enter 2024-03-29T11:13:17.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.463+0000 7fafa67fc640 10 librados: watch_flush exit 2024-03-29T11:13:17.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.463+0000 7fafa67fc640 20 client.4409.objecter shutdown clearing up homeless session... 2024-03-29T11:13:17.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.463+0000 7fafa67fc640 10 client.4409.objecter successfully canceled tick 2024-03-29T11:13:17.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.463+0000 7fafa67fc640 1 -- 172.21.15.18:0/882394337 >> v1:172.21.15.70:6816/3101871562 conn(0x7fafac0768f0 legacy=0x7fafac078db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:17.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.463+0000 7fafa67fc640 10 monclient: shutdown 2024-03-29T11:13:17.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.463+0000 7fafa67fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:17.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.463+0000 7fafa67fc640 1 -- 172.21.15.18:0/882394337 >> v1:172.21.15.18:6790/0 conn(0x7fafc01457f0 legacy=0x7fafc01db4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:17.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.463+0000 7fafa67fc640 1 -- 172.21.15.18:0/882394337 shutdown_connections 2024-03-29T11:13:17.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.463+0000 7fafa67fc640 1 -- 172.21.15.18:0/882394337 >> 172.21.15.18:0/882394337 conn(0x7fafc0066a00 msgr2=0x7fafc0136d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:17.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.463+0000 7fafa67fc640 1 -- 172.21.15.18:0/882394337 shutdown_connections 2024-03-29T11:13:17.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.463+0000 7fafa67fc640 1 -- 172.21.15.18:0/882394337 wait complete. 2024-03-29T11:13:17.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:17.463+0000 7fafa67fc640 1 librados: shutdown 2024-03-29T11:13:17.458 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:13:24.267 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:13:24.414 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.436+0000 7f852ec55640 10 monclient: get_monmap_and_config 2024-03-29T11:13:24.414 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.436+0000 7f852ec55640 10 monclient: build_initial_monmap 2024-03-29T11:13:24.414 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.436+0000 7f852ec55640 10 monclient: monmap: 2024-03-29T11:13:24.415 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:13:24.415 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:24.415 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:13:24.437573+0000 2024-03-29T11:13:24.415 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:13:24.437573+0000 2024-03-29T11:13:24.415 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:24.415 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:24.415 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:13:24.415 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:13:24.415 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:13:24.415 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:24.416 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.437+0000 7f852ec55640 1 Processor -- start 2024-03-29T11:13:24.416 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.437+0000 7f852ec55640 1 -- start start 2024-03-29T11:13:24.416 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.437+0000 7f852ec55640 10 monclient: init 2024-03-29T11:13:24.416 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.438+0000 7f852ec55640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:24.417 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.438+0000 7f852ec55640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:13:24.417 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.439+0000 7f852ec55640 10 monclient(hunting): picked mon.noname-b con 0x7f852806cac0 addr v1:172.21.15.18:6790/0 2024-03-29T11:13:24.417 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.439+0000 7f852ec55640 10 monclient(hunting): picked mon.noname-c con 0x7f852806cea0 addr v1:172.21.15.70:6789/0 2024-03-29T11:13:24.417 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.439+0000 7f852c9f1640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f852806cac0 0x7f8528071ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:32932/0 (socket says 172.21.15.18:32932) 2024-03-29T11:13:24.419 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.439+0000 7f852ec55640 10 monclient(hunting): picked mon.noname-a con 0x7f8528145e80 addr v1:172.21.15.18:6789/0 2024-03-29T11:13:24.419 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.439+0000 7f852ec55640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85280ac850 con 0x7f8528145e80 2024-03-29T11:13:24.419 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.439+0000 7f852ec55640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8528070470 con 0x7f852806cea0 2024-03-29T11:13:24.419 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.439+0000 7f852ec55640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8528072600 con 0x7f852806cac0 2024-03-29T11:13:24.419 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.439+0000 7f852ec55640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:24.419 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.439+0000 7f852ec55640 10 monclient(hunting): authenticate will time out at 1030.711435s 2024-03-29T11:13:24.419 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.439+0000 7f852c9f1640 1 -- 172.21.15.18:0/700808553 learned_addr learned my addr 172.21.15.18:0/700808553 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:13:24.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.440+0000 7f85277fe640 1 -- 172.21.15.18:0/700808553 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2782931863 0 0) 0x7f8528070470 con 0x7f852806cea0 2024-03-29T11:13:24.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.440+0000 7f85277fe640 10 monclient(hunting): _negotiate 2024-03-29T11:13:24.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.440+0000 7f85277fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:24.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.440+0000 7f85277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:24.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.440+0000 7f85277fe640 10 monclient(hunting): my global_id is 4360 2024-03-29T11:13:24.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.440+0000 7f85277fe640 1 -- 172.21.15.18:0/700808553 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8518003520 con 0x7f852806cea0 2024-03-29T11:13:24.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.440+0000 7f85277fe640 1 -- 172.21.15.18:0/700808553 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4122463120 0 0) 0x7f85280ac850 con 0x7f8528145e80 2024-03-29T11:13:24.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.440+0000 7f85277fe640 10 monclient(hunting): _negotiate 2024-03-29T11:13:24.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.440+0000 7f85277fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:24.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.440+0000 7f85277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:24.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.440+0000 7f85277fe640 10 monclient(hunting): my global_id is 4365 2024-03-29T11:13:24.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.440+0000 7f85277fe640 1 -- 172.21.15.18:0/700808553 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8528070470 con 0x7f8528145e80 2024-03-29T11:13:24.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.440+0000 7f85277fe640 1 -- 172.21.15.18:0/700808553 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 733335875 0 0) 0x7f8528072600 con 0x7f852806cac0 2024-03-29T11:13:24.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.440+0000 7f85277fe640 10 monclient(hunting): _negotiate 2024-03-29T11:13:24.421 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.440+0000 7f85277fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:24.421 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.440+0000 7f85277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:24.421 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.440+0000 7f85277fe640 10 monclient(hunting): my global_id is 4412 2024-03-29T11:13:24.421 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.440+0000 7f85277fe640 1 -- 172.21.15.18:0/700808553 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85280ac850 con 0x7f852806cac0 2024-03-29T11:13:24.421 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.440+0000 7f85277fe640 1 -- 172.21.15.18:0/700808553 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1812445466 0 0) 0x7f8528070470 con 0x7f8528145e80 2024-03-29T11:13:24.421 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.440+0000 7f85277fe640 1 -- 172.21.15.18:0/700808553 >> v1:172.21.15.18:6790/0 conn(0x7f852806cac0 legacy=0x7f8528071ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:24.421 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.440+0000 7f85277fe640 1 -- 172.21.15.18:0/700808553 >> v1:172.21.15.70:6789/0 conn(0x7f852806cea0 legacy=0x7f8528145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:24.421 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.440+0000 7f85277fe640 10 monclient: _finish_hunting 0 2024-03-29T11:13:24.421 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.440+0000 7f85277fe640 1 monclient: found mon.noname-a 2024-03-29T11:13:24.421 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.440+0000 7f85277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:24.421 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.440+0000 7f85277fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-29T11:13:24.421 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.440+0000 7f85277fe640 1 -- 172.21.15.18:0/700808553 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85281462d0 con 0x7f8528145e80 2024-03-29T11:13:24.422 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.440+0000 7f85277fe640 10 monclient: _finish_auth 0 2024-03-29T11:13:24.422 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.440+0000 7f85277fe640 10 monclient: _check_auth_tickets 2024-03-29T11:13:24.422 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.440+0000 7f85277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:24.422 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.440+0000 7f85277fe640 1 -- 172.21.15.18:0/700808553 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f85200019f0 con 0x7f8528145e80 2024-03-29T11:13:24.422 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.440+0000 7f85277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:24.422 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.440+0000 7f85277fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:13:24.422 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.440+0000 7f85277fe640 10 monclient: dump: 2024-03-29T11:13:24.422 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:13:24.422 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:24.422 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:24.422 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:24.422 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:24.422 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:24.423 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:13:24.423 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:13:24.423 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:13:24.423 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:24.423 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.441+0000 7f852ec55640 5 monclient: authenticate success, global_id 4365 2024-03-29T11:13:24.423 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.441+0000 7f852ec55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:24.423 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.441+0000 7f85277fe640 1 -- 172.21.15.18:0/700808553 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8520001b50 con 0x7f8528145e80 2024-03-29T11:13:24.423 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.441+0000 7f85277fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:24.424 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.441+0000 7f852ec55640 10 monclient: get_monmap_and_config success 2024-03-29T11:13:24.424 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.441+0000 7f852ec55640 4 set_mon_vals no callback set 2024-03-29T11:13:24.424 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.441+0000 7f85277fe640 1 -- 172.21.15.18:0/700808553 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f8520001f60 con 0x7f8528145e80 2024-03-29T11:13:24.424 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.441+0000 7f85277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:24.424 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.441+0000 7f85277fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:13:24.424 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.441+0000 7f85277fe640 10 monclient: dump: 2024-03-29T11:13:24.424 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:13:24.424 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:24.424 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:24.424 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:24.424 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:24.424 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:24.424 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:13:24.424 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:13:24.424 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:13:24.425 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:24.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.442+0000 7f852ec55640 10 monclient: shutdown 2024-03-29T11:13:24.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.442+0000 7f852ec55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:24.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.442+0000 7f852ec55640 1 -- 172.21.15.18:0/700808553 >> v1:172.21.15.18:6789/0 conn(0x7f8528145e80 legacy=0x7f852814db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:24.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.442+0000 7f852ec55640 1 -- 172.21.15.18:0/700808553 shutdown_connections 2024-03-29T11:13:24.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.442+0000 7f852ec55640 1 -- 172.21.15.18:0/700808553 >> 172.21.15.18:0/700808553 conn(0x7f85280675b0 msgr2=0x7f85280b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:24.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.443+0000 7f852ec55640 1 -- 172.21.15.18:0/700808553 shutdown_connections 2024-03-29T11:13:24.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.443+0000 7f852ec55640 1 -- 172.21.15.18:0/700808553 wait complete. 2024-03-29T11:13:24.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.444+0000 7f852ec55640 10 monclient: build_initial_monmap 2024-03-29T11:13:24.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.444+0000 7f852ec55640 10 monclient: monmap: 2024-03-29T11:13:24.432 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:13:24.432 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:13:24.432 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:13:24.433 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:13:24.433 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:24.433 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:24.433 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:13:24.433 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:13:24.433 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:13:24.433 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:24.433 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.444+0000 7f852ec55640 1 librados: starting msgr at 2024-03-29T11:13:24.433 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.444+0000 7f852ec55640 1 librados: starting objecter 2024-03-29T11:13:24.433 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.445+0000 7f852ec55640 1 Processor -- start 2024-03-29T11:13:24.433 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.445+0000 7f852ec55640 1 -- start start 2024-03-29T11:13:24.433 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.445+0000 7f852ec55640 1 librados: setting wanted keys 2024-03-29T11:13:24.433 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.445+0000 7f852ec55640 1 librados: calling monclient init 2024-03-29T11:13:24.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.445+0000 7f852ec55640 10 monclient: init 2024-03-29T11:13:24.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.446+0000 7f852ec55640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:24.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.446+0000 7f852ec55640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:13:24.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.446+0000 7f852ec55640 10 monclient(hunting): picked mon.noname-c con 0x7f852806cac0 addr v1:172.21.15.18:6790/0 2024-03-29T11:13:24.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.446+0000 7f852ec55640 10 monclient(hunting): picked mon.noname-b con 0x7f852806cea0 addr v1:172.21.15.70:6789/0 2024-03-29T11:13:24.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.446+0000 7f852c9f1640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f852806cac0 0x7f85280b3ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:33814/0 (socket says 172.21.15.18:33814) 2024-03-29T11:13:24.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.446+0000 7f852ec55640 10 monclient(hunting): picked mon.noname-a con 0x7f8528145e80 addr v1:172.21.15.18:6789/0 2024-03-29T11:13:24.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.446+0000 7f852ec55640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8528133cd0 con 0x7f8528145e80 2024-03-29T11:13:24.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.446+0000 7f852ec55640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f852813faa0 con 0x7f852806cea0 2024-03-29T11:13:24.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.446+0000 7f852ec55640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85280ab9b0 con 0x7f852806cac0 2024-03-29T11:13:24.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.446+0000 7f852ec55640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:24.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.446+0000 7f852ec55640 10 monclient(hunting): authenticate will time out at 1030.718690s 2024-03-29T11:13:24.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.446+0000 7f8527fff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f852806cea0 0x7f85281e3190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.18:42568/0 (socket says 172.21.15.18:42568) 2024-03-29T11:13:24.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.446+0000 7f8527fff640 1 -- 172.21.15.18:0/1473500467 learned_addr learned my addr 172.21.15.18:0/1473500467 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:13:24.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.447+0000 7f85257fa640 10 client.?.objecter ms_handle_connect 0x7f852806cea0 2024-03-29T11:13:24.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.447+0000 7f85257fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:24.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.447+0000 7f85257fa640 1 -- 172.21.15.18:0/1473500467 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4138313615 0 0) 0x7f852813faa0 con 0x7f852806cea0 2024-03-29T11:13:24.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.447+0000 7f85257fa640 10 monclient(hunting): _negotiate 2024-03-29T11:13:24.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.447+0000 7f85257fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:24.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.447+0000 7f85257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:24.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.447+0000 7f85257fa640 10 monclient(hunting): my global_id is 4363 2024-03-29T11:13:24.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.447+0000 7f85257fa640 1 -- 172.21.15.18:0/1473500467 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f850c004eb0 con 0x7f852806cea0 2024-03-29T11:13:24.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.447+0000 7f85257fa640 10 client.?.objecter ms_handle_connect 0x7f852806cac0 2024-03-29T11:13:24.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.447+0000 7f85257fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:24.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.447+0000 7f85257fa640 10 client.?.objecter ms_handle_connect 0x7f8528145e80 2024-03-29T11:13:24.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.447+0000 7f85257fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:24.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.447+0000 7f85257fa640 1 -- 172.21.15.18:0/1473500467 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2280525931 0 0) 0x7f85280ab9b0 con 0x7f852806cac0 2024-03-29T11:13:24.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.447+0000 7f85257fa640 10 monclient(hunting): _negotiate 2024-03-29T11:13:24.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.447+0000 7f85257fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:24.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.447+0000 7f85257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:24.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.447+0000 7f85257fa640 10 monclient(hunting): my global_id is 4415 2024-03-29T11:13:24.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.447+0000 7f85257fa640 1 -- 172.21.15.18:0/1473500467 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f852813faa0 con 0x7f852806cac0 2024-03-29T11:13:24.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.447+0000 7f85257fa640 1 -- 172.21.15.18:0/1473500467 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2055890179 0 0) 0x7f8528133cd0 con 0x7f8528145e80 2024-03-29T11:13:24.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.447+0000 7f85257fa640 10 monclient(hunting): _negotiate 2024-03-29T11:13:24.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.447+0000 7f85257fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:24.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.447+0000 7f85257fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:24.436 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.447+0000 7f85257fa640 10 monclient(hunting): my global_id is 4368 2024-03-29T11:13:24.437 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.447+0000 7f85257fa640 1 -- 172.21.15.18:0/1473500467 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85280ab9b0 con 0x7f8528145e80 2024-03-29T11:13:24.437 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.448+0000 7f85257fa640 1 -- 172.21.15.18:0/1473500467 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 903788117 0 0) 0x7f852813faa0 con 0x7f852806cac0 2024-03-29T11:13:24.437 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.448+0000 7f85257fa640 1 -- 172.21.15.18:0/1473500467 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8528133cd0 con 0x7f852806cac0 2024-03-29T11:13:24.437 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.448+0000 7f85257fa640 1 -- 172.21.15.18:0/1473500467 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f851c001930 con 0x7f852806cac0 2024-03-29T11:13:24.437 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.448+0000 7f85257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:24.437 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.448+0000 7f85257fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:13:24.437 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.448+0000 7f85257fa640 10 monclient(hunting): dump: 2024-03-29T11:13:24.437 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:13:24.437 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:24.437 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:24.437 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:24.437 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:24.437 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:24.437 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:13:24.437 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:13:24.438 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:13:24.438 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:24.438 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.448+0000 7f85257fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:13:24.438 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.448+0000 7f85257fa640 1 -- 172.21.15.18:0/1473500467 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1160047373 0 0) 0x7f85280ab9b0 con 0x7f8528145e80 2024-03-29T11:13:24.438 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.448+0000 7f85257fa640 1 -- 172.21.15.18:0/1473500467 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f852813faa0 con 0x7f8528145e80 2024-03-29T11:13:24.438 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.448+0000 7f85257fa640 1 -- 172.21.15.18:0/1473500467 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f8520001e60 con 0x7f8528145e80 2024-03-29T11:13:24.438 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.448+0000 7f85257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:24.438 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.448+0000 7f85257fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:13:24.438 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.448+0000 7f85257fa640 10 monclient(hunting): dump: 2024-03-29T11:13:24.438 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:13:24.438 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:24.438 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:24.438 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:24.438 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:24.439 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:24.439 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:13:24.439 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:13:24.439 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:13:24.439 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:24.439 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.448+0000 7f85257fa640 1 -- 172.21.15.18:0/1473500467 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3123402856 0 0) 0x7f850c004eb0 con 0x7f852806cea0 2024-03-29T11:13:24.439 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.448+0000 7f85257fa640 1 -- 172.21.15.18:0/1473500467 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85280ab9b0 con 0x7f852806cea0 2024-03-29T11:13:24.439 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.448+0000 7f85257fa640 1 -- 172.21.15.18:0/1473500467 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f8514001960 con 0x7f852806cea0 2024-03-29T11:13:24.439 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.449+0000 7f85257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:24.439 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.449+0000 7f85257fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:13:24.439 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.449+0000 7f85257fa640 10 monclient(hunting): dump: 2024-03-29T11:13:24.439 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:13:24.439 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:24.439 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:24.439 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:24.440 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:24.440 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:24.440 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:13:24.440 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:13:24.440 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:13:24.440 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:24.440 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.449+0000 7f85257fa640 1 -- 172.21.15.18:0/1473500467 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3584301566 0 0) 0x7f852813faa0 con 0x7f8528145e80 2024-03-29T11:13:24.440 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.449+0000 7f85257fa640 1 -- 172.21.15.18:0/1473500467 >> v1:172.21.15.18:6790/0 conn(0x7f852806cac0 legacy=0x7f85280b3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:24.440 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.449+0000 7f85257fa640 1 -- 172.21.15.18:0/1473500467 >> v1:172.21.15.70:6789/0 conn(0x7f852806cea0 legacy=0x7f85281e3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:24.440 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.449+0000 7f85257fa640 10 monclient: _finish_hunting 0 2024-03-29T11:13:24.440 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.449+0000 7f85257fa640 1 monclient: found mon.a 2024-03-29T11:13:24.440 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.449+0000 7f85257fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:24.440 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.449+0000 7f85257fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:13:24.440 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.449+0000 7f85257fa640 1 -- 172.21.15.18:0/1473500467 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85281361f0 con 0x7f8528145e80 2024-03-29T11:13:24.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.449+0000 7f85257fa640 10 monclient: _finish_auth 0 2024-03-29T11:13:24.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.449+0000 7f85257fa640 10 monclient: _check_auth_tickets 2024-03-29T11:13:24.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.449+0000 7f85257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:24.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.450+0000 7f85257fa640 1 -- 172.21.15.18:0/1473500467 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85200012e0 con 0x7f8528145e80 2024-03-29T11:13:24.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.450+0000 7f85257fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:24.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.450+0000 7f85257fa640 1 -- 172.21.15.18:0/1473500467 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f8520003c20 con 0x7f8528145e80 2024-03-29T11:13:24.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.450+0000 7f852ec55640 5 monclient: authenticate success, global_id 4368 2024-03-29T11:13:24.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.450+0000 7f85267fc640 4 set_mon_vals no callback set 2024-03-29T11:13:24.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.450+0000 7f852ec55640 10 monclient: _renew_subs 2024-03-29T11:13:24.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.450+0000 7f852ec55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:13:24.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.450+0000 7f852ec55640 1 -- 172.21.15.18:0/1473500467 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85280abc20 con 0x7f8528145e80 2024-03-29T11:13:24.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.450+0000 7f852ec55640 10 client.4368.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:13:24.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.450+0000 7f85257fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:24.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.450+0000 7f85257fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:13:24.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.450+0000 7f85257fa640 10 monclient: dump: 2024-03-29T11:13:24.442 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:13:24.442 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:24.442 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:24.442 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:24.442 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:24.442 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:24.442 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:13:24.442 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:13:24.442 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:13:24.442 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:24.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.450+0000 7f852ec55640 10 monclient: _renew_subs 2024-03-29T11:13:24.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.450+0000 7f852ec55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:13:24.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.450+0000 7f852ec55640 1 -- 172.21.15.18:0/1473500467 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f85280aaa00 con 0x7f8528145e80 2024-03-29T11:13:24.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.450+0000 7f852ec55640 1 librados: init done 2024-03-29T11:13:24.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.452+0000 7f852ec55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:24.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.452+0000 7f852ec55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:24.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.452+0000 7f852ec55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:13:24.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.452+0000 7f852ec55640 1 -- 172.21.15.18:0/1473500467 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8528075f10 con 0x7f8528145e80 2024-03-29T11:13:24.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.454+0000 7f85257fa640 1 -- 172.21.15.18:0/1473500467 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f8520019bf0 con 0x7f8528145e80 2024-03-29T11:13:24.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.455+0000 7f852c9f1640 10 monclient: get_auth_request con 0x7f850c06e160 auth_method 0 2024-03-29T11:13:24.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.455+0000 7f85257fa640 1 -- 172.21.15.18:0/1473500467 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f85200838b0 con 0x7f8528145e80 2024-03-29T11:13:24.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.455+0000 7f85257fa640 10 client.4368.objecter ms_dispatch 0x7f8528000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:13:24.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.455+0000 7f85257fa640 3 client.4368.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:13:24.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.455+0000 7f85257fa640 3 client.4368.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:13:24.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.455+0000 7f85257fa640 20 client.4368.objecter dump_active .. 0 homeless 2024-03-29T11:13:24.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.456+0000 7f85257fa640 10 client.4368.objecter ms_handle_connect 0x7f850c06e160 2024-03-29T11:13:24.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.461+0000 7f85257fa640 1 -- 172.21.15.18:0/1473500467 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f852004f190 con 0x7f8528145e80 2024-03-29T11:13:24.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.461+0000 7f85257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:24.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.461+0000 7f85257fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:13:24.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.858+0000 7f852ec55640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:24.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.858+0000 7f852ec55640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:24.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.858+0000 7f852ec55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:13:24.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.858+0000 7f852ec55640 1 -- 172.21.15.18:0/1473500467 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f852806d450 con 0x7f8528145e80 2024-03-29T11:13:24.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.859+0000 7f85257fa640 1 -- 172.21.15.18:0/1473500467 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f85200510c0 con 0x7f8528145e80 2024-03-29T11:13:24.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.859+0000 7f85257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:24.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.859+0000 7f85257fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:13:24.837 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:13:24.837 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:13:24.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.862+0000 7f852ec55640 10 librados: watch_flush enter 2024-03-29T11:13:24.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.862+0000 7f852ec55640 10 librados: watch_flush exit 2024-03-29T11:13:24.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.862+0000 7f852ec55640 20 client.4368.objecter shutdown clearing up homeless session... 2024-03-29T11:13:24.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.862+0000 7f852ec55640 10 client.4368.objecter successfully canceled tick 2024-03-29T11:13:24.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.862+0000 7f852ec55640 1 -- 172.21.15.18:0/1473500467 >> v1:172.21.15.70:6816/3101871562 conn(0x7f850c06e160 legacy=0x7f850c070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:24.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.862+0000 7f852ec55640 10 monclient: shutdown 2024-03-29T11:13:24.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.862+0000 7f852ec55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:24.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.862+0000 7f852ec55640 1 -- 172.21.15.18:0/1473500467 >> v1:172.21.15.18:6789/0 conn(0x7f8528145e80 legacy=0x7f85281e68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:24.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.862+0000 7f852ec55640 1 -- 172.21.15.18:0/1473500467 shutdown_connections 2024-03-29T11:13:24.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.862+0000 7f852ec55640 1 -- 172.21.15.18:0/1473500467 >> 172.21.15.18:0/1473500467 conn(0x7f85280675b0 msgr2=0x7f8528143f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:24.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.862+0000 7f852ec55640 1 -- 172.21.15.18:0/1473500467 shutdown_connections 2024-03-29T11:13:24.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.862+0000 7f852ec55640 1 -- 172.21.15.18:0/1473500467 wait complete. 2024-03-29T11:13:24.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:24.863+0000 7f852ec55640 1 librados: shutdown 2024-03-29T11:13:24.855 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:13:31.664 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:13:31.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.845+0000 7ff25359e640 10 monclient: get_monmap_and_config 2024-03-29T11:13:31.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.845+0000 7ff25359e640 10 monclient: build_initial_monmap 2024-03-29T11:13:31.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.845+0000 7ff25359e640 10 monclient: monmap: 2024-03-29T11:13:31.824 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:13:31.824 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:31.824 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:13:31.846504+0000 2024-03-29T11:13:31.824 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:13:31.846504+0000 2024-03-29T11:13:31.824 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:31.824 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:31.824 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:13:31.824 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:13:31.824 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:13:31.824 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:31.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.846+0000 7ff25359e640 1 Processor -- start 2024-03-29T11:13:31.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.846+0000 7ff25359e640 1 -- start start 2024-03-29T11:13:31.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.846+0000 7ff25359e640 10 monclient: init 2024-03-29T11:13:31.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.847+0000 7ff25359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:31.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.847+0000 7ff25359e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:13:31.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.847+0000 7ff25359e640 10 monclient(hunting): picked mon.noname-c con 0x7ff24c006890 addr v1:172.21.15.70:6789/0 2024-03-29T11:13:31.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.847+0000 7ff25359e640 10 monclient(hunting): picked mon.noname-b con 0x7ff24c006c70 addr v1:172.21.15.18:6790/0 2024-03-29T11:13:31.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.848+0000 7ff25359e640 10 monclient(hunting): picked mon.noname-a con 0x7ff24c02b4c0 addr v1:172.21.15.18:6789/0 2024-03-29T11:13:31.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.848+0000 7ff25359e640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff24c02c830 con 0x7ff24c02b4c0 2024-03-29T11:13:31.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.848+0000 7ff25359e640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff24c02afa0 con 0x7ff24c006890 2024-03-29T11:13:31.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.848+0000 7ff25359e640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff24c02aba0 con 0x7ff24c006c70 2024-03-29T11:13:31.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.848+0000 7ff25359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:31.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.848+0000 7ff25359e640 10 monclient(hunting): authenticate will time out at 1038.120027s 2024-03-29T11:13:31.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.848+0000 7ff252d9d640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7ff24c02b4c0 0x7ff24c0cf0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:43008/0 (socket says 172.21.15.18:43008) 2024-03-29T11:13:31.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.848+0000 7ff252d9d640 1 -- 172.21.15.18:0/2401583672 learned_addr learned my addr 172.21.15.18:0/2401583672 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:13:31.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.849+0000 7ff25159a640 1 -- 172.21.15.18:0/2401583672 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2452117137 0 0) 0x7ff24c02c830 con 0x7ff24c02b4c0 2024-03-29T11:13:31.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.849+0000 7ff25159a640 10 monclient(hunting): _negotiate 2024-03-29T11:13:31.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.849+0000 7ff25159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:31.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.849+0000 7ff25159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:31.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.849+0000 7ff25159a640 10 monclient(hunting): my global_id is 4371 2024-03-29T11:13:31.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.849+0000 7ff25159a640 1 -- 172.21.15.18:0/2401583672 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff23c003500 con 0x7ff24c02b4c0 2024-03-29T11:13:31.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.849+0000 7ff25159a640 1 -- 172.21.15.18:0/2401583672 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3441911619 0 0) 0x7ff23c003500 con 0x7ff24c02b4c0 2024-03-29T11:13:31.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.849+0000 7ff25159a640 1 -- 172.21.15.18:0/2401583672 >> v1:172.21.15.18:6790/0 conn(0x7ff24c006c70 legacy=0x7ff24c0c6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:13:31.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.849+0000 7ff25159a640 1 -- 172.21.15.18:0/2401583672 >> v1:172.21.15.70:6789/0 conn(0x7ff24c006890 legacy=0x7ff24c0c0a90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:13:31.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.851+0000 7ff25159a640 10 monclient: _finish_hunting 0 2024-03-29T11:13:31.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.851+0000 7ff25159a640 1 monclient: found mon.noname-a 2024-03-29T11:13:31.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.851+0000 7ff25159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:31.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.851+0000 7ff25159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-29T11:13:31.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.851+0000 7ff25159a640 1 -- 172.21.15.18:0/2401583672 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff24c02ad30 con 0x7ff24c02b4c0 2024-03-29T11:13:31.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.851+0000 7ff25159a640 10 monclient: _finish_auth 0 2024-03-29T11:13:31.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.851+0000 7ff25159a640 10 monclient: _check_auth_tickets 2024-03-29T11:13:31.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.851+0000 7ff25159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:31.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.851+0000 7ff25159a640 1 -- 172.21.15.18:0/2401583672 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7ff244004cb0 con 0x7ff24c02b4c0 2024-03-29T11:13:31.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.851+0000 7ff25159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:31.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.851+0000 7ff25159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:13:31.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.851+0000 7ff25159a640 10 monclient: dump: 2024-03-29T11:13:31.831 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:13:31.831 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:31.831 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:31.831 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:31.831 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:31.831 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:31.831 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:13:31.831 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:13:31.831 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:13:31.831 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:31.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.851+0000 7ff25359e640 5 monclient: authenticate success, global_id 4371 2024-03-29T11:13:31.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.851+0000 7ff25359e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:31.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.851+0000 7ff25159a640 1 -- 172.21.15.18:0/2401583672 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff244004f10 con 0x7ff24c02b4c0 2024-03-29T11:13:31.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.851+0000 7ff25159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:31.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.851+0000 7ff25159a640 1 -- 172.21.15.18:0/2401583672 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7ff244001430 con 0x7ff24c02b4c0 2024-03-29T11:13:31.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.851+0000 7ff25159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:31.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.851+0000 7ff25159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:13:31.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.851+0000 7ff25159a640 10 monclient: dump: 2024-03-29T11:13:31.832 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:13:31.832 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:31.832 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:31.832 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:31.832 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:31.832 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:31.832 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:13:31.833 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:13:31.833 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:13:31.833 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:31.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.851+0000 7ff25359e640 10 monclient: get_monmap_and_config success 2024-03-29T11:13:31.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.851+0000 7ff25359e640 4 set_mon_vals no callback set 2024-03-29T11:13:31.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.852+0000 7ff25359e640 10 monclient: shutdown 2024-03-29T11:13:31.835 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.852+0000 7ff25359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:31.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.852+0000 7ff25359e640 1 -- 172.21.15.18:0/2401583672 >> v1:172.21.15.18:6789/0 conn(0x7ff24c02b4c0 legacy=0x7ff24c0cf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:31.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.853+0000 7ff25359e640 1 -- 172.21.15.18:0/2401583672 shutdown_connections 2024-03-29T11:13:31.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.853+0000 7ff25359e640 1 -- 172.21.15.18:0/2401583672 >> 172.21.15.18:0/2401583672 conn(0x7ff24c0058f0 msgr2=0x7ff24c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:31.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.853+0000 7ff25359e640 1 -- 172.21.15.18:0/2401583672 shutdown_connections 2024-03-29T11:13:31.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.853+0000 7ff25359e640 1 -- 172.21.15.18:0/2401583672 wait complete. 2024-03-29T11:13:31.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.854+0000 7ff25359e640 10 monclient: build_initial_monmap 2024-03-29T11:13:31.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.854+0000 7ff25359e640 10 monclient: monmap: 2024-03-29T11:13:31.836 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:13:31.836 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:13:31.836 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:13:31.836 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:13:31.836 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:31.837 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:31.837 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:13:31.837 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:13:31.837 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:13:31.837 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:31.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.855+0000 7ff25359e640 1 librados: starting msgr at 2024-03-29T11:13:31.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.855+0000 7ff25359e640 1 librados: starting objecter 2024-03-29T11:13:31.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.855+0000 7ff25359e640 1 Processor -- start 2024-03-29T11:13:31.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.855+0000 7ff25359e640 1 -- start start 2024-03-29T11:13:31.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.855+0000 7ff25359e640 1 librados: setting wanted keys 2024-03-29T11:13:31.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.855+0000 7ff25359e640 1 librados: calling monclient init 2024-03-29T11:13:31.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.855+0000 7ff25359e640 10 monclient: init 2024-03-29T11:13:31.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.856+0000 7ff25359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:31.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.856+0000 7ff25359e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:13:31.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.856+0000 7ff25359e640 10 monclient(hunting): picked mon.noname-a con 0x7ff24c006890 addr v1:172.21.15.18:6789/0 2024-03-29T11:13:31.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.856+0000 7ff25359e640 10 monclient(hunting): picked mon.noname-c con 0x7ff24c02b4c0 addr v1:172.21.15.18:6790/0 2024-03-29T11:13:31.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.856+0000 7ff25359e640 10 monclient(hunting): picked mon.noname-b con 0x7ff24c02e020 addr v1:172.21.15.70:6789/0 2024-03-29T11:13:31.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.856+0000 7ff25359e640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff24c02e640 con 0x7ff24c006890 2024-03-29T11:13:31.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.856+0000 7ff25359e640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff24c02f080 con 0x7ff24c02e020 2024-03-29T11:13:31.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.856+0000 7ff25359e640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff24c02f360 con 0x7ff24c02b4c0 2024-03-29T11:13:31.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.856+0000 7ff25359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:31.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.856+0000 7ff25359e640 10 monclient(hunting): authenticate will time out at 1038.128302s 2024-03-29T11:13:31.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.856+0000 7ff251d9b640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7ff24c02b4c0 0x7ff24c0bce30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:55388/0 (socket says 172.21.15.18:55388) 2024-03-29T11:13:31.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.856+0000 7ff251d9b640 1 -- 172.21.15.18:0/4135648000 learned_addr learned my addr 172.21.15.18:0/4135648000 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:13:31.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.859+0000 7ff242ffd640 10 client.?.objecter ms_handle_connect 0x7ff24c02b4c0 2024-03-29T11:13:31.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.859+0000 7ff242ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:31.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.859+0000 7ff242ffd640 10 client.?.objecter ms_handle_connect 0x7ff24c006890 2024-03-29T11:13:31.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.859+0000 7ff242ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:31.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.860+0000 7ff242ffd640 10 client.?.objecter ms_handle_connect 0x7ff24c02e020 2024-03-29T11:13:31.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.860+0000 7ff242ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:31.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.860+0000 7ff242ffd640 1 -- 172.21.15.18:0/4135648000 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1242309938 0 0) 0x7ff24c02f360 con 0x7ff24c02b4c0 2024-03-29T11:13:31.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.860+0000 7ff242ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:13:31.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.860+0000 7ff242ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:31.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.860+0000 7ff242ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:31.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.860+0000 7ff242ffd640 10 monclient(hunting): my global_id is 4418 2024-03-29T11:13:31.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.860+0000 7ff242ffd640 1 -- 172.21.15.18:0/4135648000 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff234004eb0 con 0x7ff24c02b4c0 2024-03-29T11:13:31.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.860+0000 7ff242ffd640 1 -- 172.21.15.18:0/4135648000 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1669304538 0 0) 0x7ff24c02f080 con 0x7ff24c02e020 2024-03-29T11:13:31.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.860+0000 7ff242ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:13:31.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.860+0000 7ff242ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:31.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.860+0000 7ff242ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:31.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.860+0000 7ff242ffd640 10 monclient(hunting): my global_id is 4366 2024-03-29T11:13:31.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.860+0000 7ff242ffd640 1 -- 172.21.15.18:0/4135648000 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff24c02f360 con 0x7ff24c02e020 2024-03-29T11:13:31.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.860+0000 7ff242ffd640 1 -- 172.21.15.18:0/4135648000 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2199438683 0 0) 0x7ff248000ff0 con 0x7ff24c006890 2024-03-29T11:13:31.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.860+0000 7ff242ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:13:31.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.860+0000 7ff242ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:31.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.860+0000 7ff242ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:31.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.860+0000 7ff242ffd640 10 monclient(hunting): my global_id is 4374 2024-03-29T11:13:31.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.860+0000 7ff242ffd640 1 -- 172.21.15.18:0/4135648000 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff24c02f080 con 0x7ff24c006890 2024-03-29T11:13:31.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.860+0000 7ff242ffd640 1 -- 172.21.15.18:0/4135648000 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 140581752 0 0) 0x7ff24c02f360 con 0x7ff24c02e020 2024-03-29T11:13:31.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.861+0000 7ff242ffd640 1 -- 172.21.15.18:0/4135648000 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff248000ff0 con 0x7ff24c02e020 2024-03-29T11:13:31.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.861+0000 7ff242ffd640 1 -- 172.21.15.18:0/4135648000 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 44420066 0 0) 0x7ff234004eb0 con 0x7ff24c02b4c0 2024-03-29T11:13:31.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.861+0000 7ff242ffd640 1 -- 172.21.15.18:0/4135648000 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff24c02f360 con 0x7ff24c02b4c0 2024-03-29T11:13:31.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.861+0000 7ff242ffd640 1 -- 172.21.15.18:0/4135648000 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7ff244001210 con 0x7ff24c02e020 2024-03-29T11:13:31.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.861+0000 7ff242ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:31.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.861+0000 7ff242ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:13:31.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.861+0000 7ff242ffd640 10 monclient(hunting): dump: 2024-03-29T11:13:31.848 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:13:31.848 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:31.848 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:31.848 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:31.848 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:31.848 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:31.848 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:13:31.848 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:13:31.848 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:13:31.848 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:31.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.861+0000 7ff242ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:13:31.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.861+0000 7ff242ffd640 1 -- 172.21.15.18:0/4135648000 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7ff25404fdd0 con 0x7ff24c02b4c0 2024-03-29T11:13:31.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.861+0000 7ff242ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:31.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.861+0000 7ff242ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:13:31.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.861+0000 7ff242ffd640 10 monclient(hunting): dump: 2024-03-29T11:13:31.849 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:13:31.849 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:31.849 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:31.849 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:31.849 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:31.849 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:31.849 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:13:31.849 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:13:31.850 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:13:31.850 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:31.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.861+0000 7ff242ffd640 1 -- 172.21.15.18:0/4135648000 <== mon.1 v1:172.21.15.70:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1072633975 0 0) 0x7ff248000ff0 con 0x7ff24c02e020 2024-03-29T11:13:31.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.861+0000 7ff242ffd640 1 -- 172.21.15.18:0/4135648000 >> v1:172.21.15.18:6790/0 conn(0x7ff24c02b4c0 legacy=0x7ff24c0bce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:31.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.861+0000 7ff242ffd640 1 -- 172.21.15.18:0/4135648000 >> v1:172.21.15.18:6789/0 conn(0x7ff24c006890 legacy=0x7ff24c02d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:31.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.861+0000 7ff242ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:13:31.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.861+0000 7ff242ffd640 1 monclient: found mon.b 2024-03-29T11:13:31.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.861+0000 7ff242ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:31.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.861+0000 7ff242ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:13:31.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.861+0000 7ff242ffd640 1 -- 172.21.15.18:0/4135648000 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff24c0bd540 con 0x7ff24c02e020 2024-03-29T11:13:31.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.861+0000 7ff242ffd640 10 monclient: _finish_auth 0 2024-03-29T11:13:31.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.861+0000 7ff242ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:13:31.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.861+0000 7ff242ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:31.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.861+0000 7ff25359e640 5 monclient: authenticate success, global_id 4366 2024-03-29T11:13:31.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.861+0000 7ff25359e640 10 monclient: _renew_subs 2024-03-29T11:13:31.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.861+0000 7ff25359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:13:31.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.861+0000 7ff25359e640 1 -- 172.21.15.18:0/4135648000 --> v1:172.21.15.70:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff24c0bd7b0 con 0x7ff24c02e020 2024-03-29T11:13:31.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.861+0000 7ff25359e640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:13:31.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.861+0000 7ff25359e640 10 monclient: _renew_subs 2024-03-29T11:13:31.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.861+0000 7ff25359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:13:31.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.861+0000 7ff25359e640 1 -- 172.21.15.18:0/4135648000 --> v1:172.21.15.70:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff24c0c33c0 con 0x7ff24c02e020 2024-03-29T11:13:31.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.861+0000 7ff25359e640 1 librados: init done 2024-03-29T11:13:31.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.862+0000 7ff242ffd640 1 -- 172.21.15.18:0/4135648000 <== mon.1 v1:172.21.15.70:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff244001370 con 0x7ff24c02e020 2024-03-29T11:13:31.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.862+0000 7ff242ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:31.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.862+0000 7ff242ffd640 1 -- 172.21.15.18:0/4135648000 <== mon.1 v1:172.21.15.70:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7ff244004870 con 0x7ff24c02e020 2024-03-29T11:13:31.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.862+0000 7ff242ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:31.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.862+0000 7ff242ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:13:31.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.862+0000 7ff242ffd640 10 monclient: dump: 2024-03-29T11:13:31.852 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:13:31.852 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:31.852 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:31.852 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:31.852 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:31.852 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:31.852 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:13:31.852 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:13:31.852 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:13:31.853 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:31.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.862+0000 7ff250d99640 4 set_mon_vals no callback set 2024-03-29T11:13:31.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.865+0000 7ff25359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:31.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.865+0000 7ff242ffd640 1 -- 172.21.15.18:0/4135648000 <== mon.1 v1:172.21.15.70:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7ff244001b40 con 0x7ff24c02e020 2024-03-29T11:13:31.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.865+0000 7ff25359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:31.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.865+0000 7ff25359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:13:31.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.865+0000 7ff25359e640 1 -- 172.21.15.18:0/4135648000 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff238005080 con 0x7ff24c02e020 2024-03-29T11:13:31.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.865+0000 7ff242ffd640 1 -- 172.21.15.18:0/4135648000 <== mon.1 v1:172.21.15.70:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7ff244084b40 con 0x7ff24c02e020 2024-03-29T11:13:31.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.865+0000 7ff242ffd640 10 client.4366.objecter ms_dispatch 0x7ff254000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:13:31.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.865+0000 7ff242ffd640 3 client.4366.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:13:31.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.865+0000 7ff242ffd640 3 client.4366.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:13:31.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.865+0000 7ff242ffd640 20 client.4366.objecter dump_active .. 0 homeless 2024-03-29T11:13:31.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.866+0000 7ff25259c640 10 monclient: get_auth_request con 0x7ff23406e0d0 auth_method 0 2024-03-29T11:13:31.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.866+0000 7ff242ffd640 10 client.4366.objecter ms_handle_connect 0x7ff23406e0d0 2024-03-29T11:13:31.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.872+0000 7ff242ffd640 1 -- 172.21.15.18:0/4135648000 <== mon.1 v1:172.21.15.70:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff244050420 con 0x7ff24c02e020 2024-03-29T11:13:31.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.872+0000 7ff242ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:31.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:31.872+0000 7ff242ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:13:32.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:32.270+0000 7ff25359e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:32.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:32.270+0000 7ff25359e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:32.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:32.270+0000 7ff25359e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:13:32.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:32.270+0000 7ff25359e640 1 -- 172.21.15.18:0/4135648000 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff238005330 con 0x7ff24c02e020 2024-03-29T11:13:32.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:32.271+0000 7ff242ffd640 1 -- 172.21.15.18:0/4135648000 <== mon.1 v1:172.21.15.70:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7ff244052350 con 0x7ff24c02e020 2024-03-29T11:13:32.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:32.271+0000 7ff242ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:32.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:32.271+0000 7ff242ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:13:32.250 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:13:32.250 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:13:32.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:32.275+0000 7ff25359e640 10 librados: watch_flush enter 2024-03-29T11:13:32.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:32.276+0000 7ff25359e640 10 librados: watch_flush exit 2024-03-29T11:13:32.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:32.276+0000 7ff25359e640 20 client.4366.objecter shutdown clearing up homeless session... 2024-03-29T11:13:32.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:32.276+0000 7ff25359e640 10 client.4366.objecter successfully canceled tick 2024-03-29T11:13:32.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:32.276+0000 7ff25359e640 1 -- 172.21.15.18:0/4135648000 >> v1:172.21.15.70:6816/3101871562 conn(0x7ff23406e0d0 legacy=0x7ff234070590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:32.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:32.276+0000 7ff25359e640 10 monclient: shutdown 2024-03-29T11:13:32.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:32.276+0000 7ff25359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:32.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:32.276+0000 7ff25359e640 1 -- 172.21.15.18:0/4135648000 >> v1:172.21.15.70:6789/0 conn(0x7ff24c02e020 legacy=0x7ff24c0c2ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:32.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:32.276+0000 7ff25359e640 1 -- 172.21.15.18:0/4135648000 shutdown_connections 2024-03-29T11:13:32.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:32.276+0000 7ff25359e640 1 -- 172.21.15.18:0/4135648000 >> 172.21.15.18:0/4135648000 conn(0x7ff24c0058f0 msgr2=0x7ff24c038130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:32.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:32.276+0000 7ff25359e640 1 -- 172.21.15.18:0/4135648000 shutdown_connections 2024-03-29T11:13:32.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:32.277+0000 7ff25359e640 1 -- 172.21.15.18:0/4135648000 wait complete. 2024-03-29T11:13:32.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:32.277+0000 7ff25359e640 1 librados: shutdown 2024-03-29T11:13:32.270 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:13:39.079 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:13:39.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.251+0000 7f310c0c8640 10 monclient: get_monmap_and_config 2024-03-29T11:13:39.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.251+0000 7f310c0c8640 10 monclient: build_initial_monmap 2024-03-29T11:13:39.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.251+0000 7f310c0c8640 10 monclient: monmap: 2024-03-29T11:13:39.229 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:13:39.230 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:39.230 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:13:39.252718+0000 2024-03-29T11:13:39.230 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:13:39.252718+0000 2024-03-29T11:13:39.230 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:39.230 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:39.230 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:13:39.230 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:13:39.230 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:13:39.230 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:39.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.252+0000 7f310c0c8640 1 Processor -- start 2024-03-29T11:13:39.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.252+0000 7f310c0c8640 1 -- start start 2024-03-29T11:13:39.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.253+0000 7f310c0c8640 10 monclient: init 2024-03-29T11:13:39.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.253+0000 7f310c0c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:39.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.253+0000 7f310c0c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:13:39.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.254+0000 7f310c0c8640 10 monclient(hunting): picked mon.noname-b con 0x7f310406ab70 addr v1:172.21.15.18:6790/0 2024-03-29T11:13:39.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.254+0000 7f310c0c8640 10 monclient(hunting): picked mon.noname-a con 0x7f310406af50 addr v1:172.21.15.18:6789/0 2024-03-29T11:13:39.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.254+0000 7f310c0c8640 10 monclient(hunting): picked mon.noname-c con 0x7f3104141fb0 addr v1:172.21.15.70:6789/0 2024-03-29T11:13:39.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.254+0000 7f310c0c8640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31040b0d50 con 0x7f310406af50 2024-03-29T11:13:39.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.254+0000 7f310c0c8640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31040b1030 con 0x7f3104141fb0 2024-03-29T11:13:39.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.254+0000 7f310c0c8640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31040afa70 con 0x7f310406ab70 2024-03-29T11:13:39.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.254+0000 7f310c0c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:39.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.254+0000 7f310c0c8640 10 monclient(hunting): authenticate will time out at 1045.526013s 2024-03-29T11:13:39.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.254+0000 7f3109663640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f310406af50 0x7f3104072290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:47014/0 (socket says 172.21.15.18:47014) 2024-03-29T11:13:39.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.254+0000 7f3109663640 1 -- 172.21.15.18:0/1162331424 learned_addr learned my addr 172.21.15.18:0/1162331424 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:13:39.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.254+0000 7f3108e62640 1 -- 172.21.15.18:0/1162331424 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4078648184 0 0) 0x7f31040afa70 con 0x7f310406ab70 2024-03-29T11:13:39.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.254+0000 7f3108e62640 10 monclient(hunting): _negotiate 2024-03-29T11:13:39.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.254+0000 7f3108e62640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:39.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.254+0000 7f3108e62640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:39.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.254+0000 7f3108e62640 10 monclient(hunting): my global_id is 4421 2024-03-29T11:13:39.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.254+0000 7f3108e62640 1 -- 172.21.15.18:0/1162331424 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30f0003520 con 0x7f310406ab70 2024-03-29T11:13:39.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.255+0000 7f3108e62640 1 -- 172.21.15.18:0/1162331424 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3590337861 0 0) 0x7f31040b0d50 con 0x7f310406af50 2024-03-29T11:13:39.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.255+0000 7f3108e62640 10 monclient(hunting): _negotiate 2024-03-29T11:13:39.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.255+0000 7f3108e62640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:39.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.255+0000 7f3108e62640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:39.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.255+0000 7f3108e62640 10 monclient(hunting): my global_id is 4377 2024-03-29T11:13:39.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.255+0000 7f3108e62640 1 -- 172.21.15.18:0/1162331424 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31040afa70 con 0x7f310406af50 2024-03-29T11:13:39.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.255+0000 7f3108e62640 1 -- 172.21.15.18:0/1162331424 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1812008153 0 0) 0x7f30f0003520 con 0x7f310406ab70 2024-03-29T11:13:39.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.255+0000 7f3108e62640 1 -- 172.21.15.18:0/1162331424 >> v1:172.21.15.70:6789/0 conn(0x7f3104141fb0 legacy=0x7f31040715b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:39.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.255+0000 7f3108e62640 1 -- 172.21.15.18:0/1162331424 >> v1:172.21.15.18:6789/0 conn(0x7f310406af50 legacy=0x7f3104072290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:39.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.255+0000 7f3108e62640 10 monclient: _finish_hunting 0 2024-03-29T11:13:39.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.255+0000 7f3108e62640 1 monclient: found mon.noname-b 2024-03-29T11:13:39.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.255+0000 7f3108e62640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:39.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.255+0000 7f3108e62640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:13:39.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.255+0000 7f3108e62640 1 -- 172.21.15.18:0/1162331424 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31041423e0 con 0x7f310406ab70 2024-03-29T11:13:39.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.255+0000 7f3108e62640 10 monclient: _finish_auth 0 2024-03-29T11:13:39.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.255+0000 7f3108e62640 10 monclient: _check_auth_tickets 2024-03-29T11:13:39.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.255+0000 7f3108e62640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:39.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.255+0000 7f310c0c8640 5 monclient: authenticate success, global_id 4421 2024-03-29T11:13:39.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.255+0000 7f3108e62640 1 -- 172.21.15.18:0/1162331424 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f3100001690 con 0x7f310406ab70 2024-03-29T11:13:39.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.255+0000 7f3108e62640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:39.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.255+0000 7f3108e62640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:13:39.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.255+0000 7f3108e62640 10 monclient: dump: 2024-03-29T11:13:39.236 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:13:39.236 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:39.236 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:39.236 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:39.236 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:39.237 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:39.237 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:13:39.237 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:13:39.237 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:13:39.237 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:39.237 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.255+0000 7f310c0c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:39.237 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.255+0000 7f3108e62640 1 -- 172.21.15.18:0/1162331424 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f31000017f0 con 0x7f310406ab70 2024-03-29T11:13:39.237 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.255+0000 7f3108e62640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:39.237 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.255+0000 7f310c0c8640 10 monclient: get_monmap_and_config success 2024-03-29T11:13:39.237 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.255+0000 7f310c0c8640 4 set_mon_vals no callback set 2024-03-29T11:13:39.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.256+0000 7f310c0c8640 10 monclient: shutdown 2024-03-29T11:13:39.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.256+0000 7f310c0c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:39.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.256+0000 7f310c0c8640 1 -- 172.21.15.18:0/1162331424 >> v1:172.21.15.18:6790/0 conn(0x7f310406ab70 legacy=0x7f31040af2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:39.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.257+0000 7f310c0c8640 1 -- 172.21.15.18:0/1162331424 shutdown_connections 2024-03-29T11:13:39.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.257+0000 7f310c0c8640 1 -- 172.21.15.18:0/1162331424 >> 172.21.15.18:0/1162331424 conn(0x7f3104067470 msgr2=0x7f31040b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:39.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.257+0000 7f310c0c8640 1 -- 172.21.15.18:0/1162331424 shutdown_connections 2024-03-29T11:13:39.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.257+0000 7f310c0c8640 1 -- 172.21.15.18:0/1162331424 wait complete. 2024-03-29T11:13:39.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.258+0000 7f310c0c8640 10 monclient: build_initial_monmap 2024-03-29T11:13:39.243 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.258+0000 7f310c0c8640 10 monclient: monmap: 2024-03-29T11:13:39.243 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:13:39.243 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:13:39.243 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:13:39.243 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:13:39.243 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:39.243 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:39.243 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:13:39.243 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:13:39.243 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:13:39.244 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:39.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.258+0000 7f310c0c8640 1 librados: starting msgr at 2024-03-29T11:13:39.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.258+0000 7f310c0c8640 1 librados: starting objecter 2024-03-29T11:13:39.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.259+0000 7f310c0c8640 1 Processor -- start 2024-03-29T11:13:39.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.259+0000 7f310c0c8640 1 -- start start 2024-03-29T11:13:39.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.259+0000 7f310c0c8640 1 librados: setting wanted keys 2024-03-29T11:13:39.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.259+0000 7f310c0c8640 1 librados: calling monclient init 2024-03-29T11:13:39.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.259+0000 7f310c0c8640 10 monclient: init 2024-03-29T11:13:39.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.260+0000 7f310c0c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:39.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.260+0000 7f310c0c8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:13:39.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.260+0000 7f310c0c8640 10 monclient(hunting): picked mon.noname-a con 0x7f310406ab70 addr v1:172.21.15.18:6789/0 2024-03-29T11:13:39.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.260+0000 7f310c0c8640 10 monclient(hunting): picked mon.noname-b con 0x7f3104141fb0 addr v1:172.21.15.70:6789/0 2024-03-29T11:13:39.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.260+0000 7f310c0c8640 10 monclient(hunting): picked mon.noname-c con 0x7f310413e1a0 addr v1:172.21.15.18:6790/0 2024-03-29T11:13:39.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.260+0000 7f310c0c8640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f310411d7e0 con 0x7f310406ab70 2024-03-29T11:13:39.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.260+0000 7f310c0c8640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31041dec50 con 0x7f3104141fb0 2024-03-29T11:13:39.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.260+0000 7f310c0c8640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31040ae430 con 0x7f310413e1a0 2024-03-29T11:13:39.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.260+0000 7f310a665640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f310413e1a0 0x7f31041e60a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:59504/0 (socket says 172.21.15.18:59504) 2024-03-29T11:13:39.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.260+0000 7f310a665640 1 -- 172.21.15.18:0/1851379611 learned_addr learned my addr 172.21.15.18:0/1851379611 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:13:39.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.260+0000 7f310c0c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:39.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.260+0000 7f310c0c8640 10 monclient(hunting): authenticate will time out at 1045.532270s 2024-03-29T11:13:39.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.260+0000 7f30fa7fc640 10 client.?.objecter ms_handle_connect 0x7f310413e1a0 2024-03-29T11:13:39.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.260+0000 7f30fa7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:39.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.260+0000 7f30fa7fc640 1 -- 172.21.15.18:0/1851379611 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 430917884 0 0) 0x7f31040ae430 con 0x7f310413e1a0 2024-03-29T11:13:39.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.260+0000 7f30fa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:13:39.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.260+0000 7f30fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:39.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.260+0000 7f30fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:39.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.260+0000 7f30fa7fc640 10 monclient(hunting): my global_id is 4424 2024-03-29T11:13:39.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.260+0000 7f30fa7fc640 1 -- 172.21.15.18:0/1851379611 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30e8004eb0 con 0x7f310413e1a0 2024-03-29T11:13:39.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.260+0000 7f30fa7fc640 10 client.?.objecter ms_handle_connect 0x7f310406ab70 2024-03-29T11:13:39.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.260+0000 7f30fa7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:39.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.261+0000 7f30fa7fc640 1 -- 172.21.15.18:0/1851379611 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1829006626 0 0) 0x7f310411d7e0 con 0x7f310406ab70 2024-03-29T11:13:39.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.261+0000 7f30fa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:13:39.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.261+0000 7f30fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:39.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.261+0000 7f30fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:39.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.261+0000 7f30fa7fc640 10 monclient(hunting): my global_id is 4380 2024-03-29T11:13:39.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.261+0000 7f30fa7fc640 1 -- 172.21.15.18:0/1851379611 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31040ae430 con 0x7f310406ab70 2024-03-29T11:13:39.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.261+0000 7f30fa7fc640 1 -- 172.21.15.18:0/1851379611 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2362920206 0 0) 0x7f30e8004eb0 con 0x7f310413e1a0 2024-03-29T11:13:39.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.261+0000 7f30fa7fc640 1 -- 172.21.15.18:0/1851379611 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f310411d7e0 con 0x7f310413e1a0 2024-03-29T11:13:39.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.261+0000 7f30fa7fc640 1 -- 172.21.15.18:0/1851379611 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1583325668 0 0) 0x7f31040ae430 con 0x7f310406ab70 2024-03-29T11:13:39.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.261+0000 7f30fa7fc640 1 -- 172.21.15.18:0/1851379611 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30e8004eb0 con 0x7f310406ab70 2024-03-29T11:13:39.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.261+0000 7f30fa7fc640 1 -- 172.21.15.18:0/1851379611 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f30f4001910 con 0x7f310413e1a0 2024-03-29T11:13:39.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.261+0000 7f30fa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:39.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.261+0000 7f30fa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:13:39.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.261+0000 7f30fa7fc640 10 monclient(hunting): dump: 2024-03-29T11:13:39.247 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:13:39.247 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:39.247 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:39.247 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:39.247 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:39.247 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:39.247 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:13:39.247 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:13:39.247 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:13:39.247 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:39.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.261+0000 7f30fa7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:13:39.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.261+0000 7f30fa7fc640 1 -- 172.21.15.18:0/1851379611 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f3100003790 con 0x7f310406ab70 2024-03-29T11:13:39.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.261+0000 7f30fa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:39.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.262+0000 7f30fa7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:13:39.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.262+0000 7f30fa7fc640 10 monclient(hunting): dump: 2024-03-29T11:13:39.248 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:13:39.248 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:39.248 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:39.248 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:39.248 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:39.248 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:39.248 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:13:39.248 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:13:39.248 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:13:39.248 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:39.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.262+0000 7f30fa7fc640 1 -- 172.21.15.18:0/1851379611 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1922644098 0 0) 0x7f30e8004eb0 con 0x7f310406ab70 2024-03-29T11:13:39.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.262+0000 7f30fa7fc640 1 -- 172.21.15.18:0/1851379611 >> v1:172.21.15.18:6790/0 conn(0x7f310413e1a0 legacy=0x7f31041e60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:39.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.262+0000 7f30fa7fc640 1 -- 172.21.15.18:0/1851379611 >> v1:172.21.15.70:6789/0 conn(0x7f3104141fb0 legacy=0x7f31041df170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:39.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.262+0000 7f30fa7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:13:39.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.262+0000 7f30fa7fc640 1 monclient: found mon.a 2024-03-29T11:13:39.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.262+0000 7f30fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:39.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.262+0000 7f30fa7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:13:39.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.262+0000 7f30fa7fc640 1 -- 172.21.15.18:0/1851379611 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31041df880 con 0x7f310406ab70 2024-03-29T11:13:39.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.262+0000 7f30fa7fc640 10 monclient: _finish_auth 0 2024-03-29T11:13:39.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.262+0000 7f30fa7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:13:39.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.262+0000 7f30fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:39.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.262+0000 7f310c0c8640 5 monclient: authenticate success, global_id 4380 2024-03-29T11:13:39.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.262+0000 7f310c0c8640 10 monclient: _renew_subs 2024-03-29T11:13:39.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.262+0000 7f310c0c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:13:39.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.262+0000 7f310c0c8640 1 -- 172.21.15.18:0/1851379611 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31040ad790 con 0x7f310406ab70 2024-03-29T11:13:39.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.262+0000 7f310c0c8640 10 client.4380.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:13:39.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.262+0000 7f310c0c8640 10 monclient: _renew_subs 2024-03-29T11:13:39.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.262+0000 7f310c0c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:13:39.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.262+0000 7f310c0c8640 1 -- 172.21.15.18:0/1851379611 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f31041e77c0 con 0x7f310406ab70 2024-03-29T11:13:39.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.262+0000 7f310c0c8640 1 librados: init done 2024-03-29T11:13:39.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.263+0000 7f30fa7fc640 1 -- 172.21.15.18:0/1851379611 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3100001240 con 0x7f310406ab70 2024-03-29T11:13:39.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.263+0000 7f30fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:39.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.263+0000 7f30fa7fc640 1 -- 172.21.15.18:0/1851379611 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f3100001c00 con 0x7f310406ab70 2024-03-29T11:13:39.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.263+0000 7f30fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:39.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.263+0000 7f30fa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:13:39.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.263+0000 7f30fa7fc640 10 monclient: dump: 2024-03-29T11:13:39.251 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:13:39.251 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:39.251 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:39.251 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:39.251 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:39.251 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:39.251 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:13:39.251 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:13:39.252 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:13:39.252 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:39.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.264+0000 7f30fbfff640 4 set_mon_vals no callback set 2024-03-29T11:13:39.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.264+0000 7f310c0c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:39.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.265+0000 7f310c0c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:39.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.265+0000 7f310c0c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:13:39.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.265+0000 7f310c0c8640 1 -- 172.21.15.18:0/1851379611 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31041deab0 con 0x7f310406ab70 2024-03-29T11:13:39.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.265+0000 7f30fa7fc640 1 -- 172.21.15.18:0/1851379611 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f3100019a10 con 0x7f310406ab70 2024-03-29T11:13:39.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.266+0000 7f30fa7fc640 1 -- 172.21.15.18:0/1851379611 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f3100083940 con 0x7f310406ab70 2024-03-29T11:13:39.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.266+0000 7f30fa7fc640 10 client.4380.objecter ms_dispatch 0x7f3104000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:13:39.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.266+0000 7f30fa7fc640 3 client.4380.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:13:39.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.266+0000 7f30fa7fc640 3 client.4380.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:13:39.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.266+0000 7f30fa7fc640 20 client.4380.objecter dump_active .. 0 homeless 2024-03-29T11:13:39.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.266+0000 7f3109663640 10 monclient: get_auth_request con 0x7f30e806dee0 auth_method 0 2024-03-29T11:13:39.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.266+0000 7f30fa7fc640 10 client.4380.objecter ms_handle_connect 0x7f30e806dee0 2024-03-29T11:13:39.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.276+0000 7f30fa7fc640 1 -- 172.21.15.18:0/1851379611 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f31041deab0 con 0x7f310406ab70 2024-03-29T11:13:39.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.276+0000 7f30fa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:39.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.276+0000 7f30fa7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:13:39.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.606+0000 7f310c0c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:39.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.606+0000 7f310c0c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:39.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.606+0000 7f310c0c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:13:39.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.606+0000 7f310c0c8640 1 -- 172.21.15.18:0/1851379611 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31041deab0 con 0x7f310406ab70 2024-03-29T11:13:39.585 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.607+0000 7f30fa7fc640 1 -- 172.21.15.18:0/1851379611 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f31041deab0 con 0x7f310406ab70 2024-03-29T11:13:39.585 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.607+0000 7f30fa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:39.585 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.607+0000 7f30fa7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:13:39.585 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:13:39.585 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:13:39.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.610+0000 7f310c0c8640 10 librados: watch_flush enter 2024-03-29T11:13:39.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.610+0000 7f310c0c8640 10 librados: watch_flush exit 2024-03-29T11:13:39.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.610+0000 7f310c0c8640 20 client.4380.objecter shutdown clearing up homeless session... 2024-03-29T11:13:39.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.610+0000 7f310c0c8640 10 client.4380.objecter successfully canceled tick 2024-03-29T11:13:39.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.610+0000 7f310c0c8640 1 -- 172.21.15.18:0/1851379611 >> v1:172.21.15.70:6816/3101871562 conn(0x7f30e806dee0 legacy=0x7f30e80703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:39.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.610+0000 7f310c0c8640 10 monclient: shutdown 2024-03-29T11:13:39.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.610+0000 7f310c0c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:39.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.610+0000 7f310c0c8640 1 -- 172.21.15.18:0/1851379611 >> v1:172.21.15.18:6789/0 conn(0x7f310406ab70 legacy=0x7f31040add20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:39.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.611+0000 7f310c0c8640 1 -- 172.21.15.18:0/1851379611 shutdown_connections 2024-03-29T11:13:39.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.611+0000 7f310c0c8640 1 -- 172.21.15.18:0/1851379611 >> 172.21.15.18:0/1851379611 conn(0x7f3104067470 msgr2=0x7f3104071520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:39.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.611+0000 7f310c0c8640 1 -- 172.21.15.18:0/1851379611 shutdown_connections 2024-03-29T11:13:39.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.611+0000 7f310c0c8640 1 -- 172.21.15.18:0/1851379611 wait complete. 2024-03-29T11:13:39.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:39.611+0000 7f310c0c8640 1 librados: shutdown 2024-03-29T11:13:39.604 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:13:46.410 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:13:46.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.588+0000 7f6097227640 10 monclient: get_monmap_and_config 2024-03-29T11:13:46.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.588+0000 7f6097227640 10 monclient: build_initial_monmap 2024-03-29T11:13:46.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.588+0000 7f6097227640 10 monclient: monmap: 2024-03-29T11:13:46.570 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:13:46.571 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:46.571 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:13:46.590313+0000 2024-03-29T11:13:46.571 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:13:46.590313+0000 2024-03-29T11:13:46.571 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:46.571 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:46.571 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:13:46.571 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:13:46.571 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:13:46.571 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:46.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.589+0000 7f6097227640 1 Processor -- start 2024-03-29T11:13:46.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.589+0000 7f6097227640 1 -- start start 2024-03-29T11:13:46.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.589+0000 7f6097227640 10 monclient: init 2024-03-29T11:13:46.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.590+0000 7f6097227640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:46.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.590+0000 7f6097227640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:13:46.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.590+0000 7f6097227640 10 monclient(hunting): picked mon.noname-a con 0x7f60900af6c0 addr v1:172.21.15.18:6789/0 2024-03-29T11:13:46.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.590+0000 7f6097227640 10 monclient(hunting): picked mon.noname-c con 0x7f609014f6b0 addr v1:172.21.15.70:6789/0 2024-03-29T11:13:46.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.590+0000 7f6097227640 10 monclient(hunting): picked mon.noname-b con 0x7f6090152980 addr v1:172.21.15.18:6790/0 2024-03-29T11:13:46.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.590+0000 7f6097227640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60900a94e0 con 0x7f60900af6c0 2024-03-29T11:13:46.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.590+0000 7f6097227640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60900afb60 con 0x7f609014f6b0 2024-03-29T11:13:46.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.590+0000 7f6097227640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60900aab80 con 0x7f6090152980 2024-03-29T11:13:46.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.590+0000 7f6097227640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:46.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.590+0000 7f6097227640 10 monclient(hunting): authenticate will time out at 1052.863377s 2024-03-29T11:13:46.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.596+0000 7f6094fc3640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f60900af6c0 0x7f60900ad450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:58941/0 (socket says 172.21.15.18:58941) 2024-03-29T11:13:46.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.596+0000 7f6094fc3640 1 -- 172.21.15.18:0/883275139 learned_addr learned my addr 172.21.15.18:0/883275139 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:13:46.579 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.599+0000 7f608f7fe640 1 -- 172.21.15.18:0/883275139 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 383676370 0 0) 0x7f60900afb60 con 0x7f609014f6b0 2024-03-29T11:13:46.579 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.599+0000 7f608f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:13:46.579 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.599+0000 7f608f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:46.579 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.599+0000 7f608f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:46.579 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.599+0000 7f608f7fe640 10 monclient(hunting): my global_id is 4369 2024-03-29T11:13:46.579 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.599+0000 7f608f7fe640 1 -- 172.21.15.18:0/883275139 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6078003520 con 0x7f609014f6b0 2024-03-29T11:13:46.579 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.599+0000 7f608f7fe640 1 -- 172.21.15.18:0/883275139 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2092555985 0 0) 0x7f6078003520 con 0x7f609014f6b0 2024-03-29T11:13:46.579 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.599+0000 7f608f7fe640 1 -- 172.21.15.18:0/883275139 >> v1:172.21.15.18:6790/0 conn(0x7f6090152980 legacy=0x7f6090152e10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:13:46.579 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.600+0000 7f608f7fe640 1 -- 172.21.15.18:0/883275139 >> v1:172.21.15.18:6789/0 conn(0x7f60900af6c0 legacy=0x7f60900ad450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:46.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.600+0000 7f608f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:13:46.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.600+0000 7f608f7fe640 1 monclient: found mon.noname-c 2024-03-29T11:13:46.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.600+0000 7f608f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:46.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.600+0000 7f608f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.70:6789/0 2024-03-29T11:13:46.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.600+0000 7f608f7fe640 1 -- 172.21.15.18:0/883275139 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60900aa680 con 0x7f609014f6b0 2024-03-29T11:13:46.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.600+0000 7f608f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:13:46.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.600+0000 7f608f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:13:46.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.600+0000 7f608f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:46.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.600+0000 7f608f7fe640 1 -- 172.21.15.18:0/883275139 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f6080001670 con 0x7f609014f6b0 2024-03-29T11:13:46.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.600+0000 7f608f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:46.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.600+0000 7f608f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:13:46.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.600+0000 7f608f7fe640 10 monclient: dump: 2024-03-29T11:13:46.583 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:13:46.583 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:46.583 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:46.583 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:46.583 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:46.583 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:46.583 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:13:46.583 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:13:46.583 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:13:46.583 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:46.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.600+0000 7f6097227640 5 monclient: authenticate success, global_id 4369 2024-03-29T11:13:46.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.600+0000 7f6097227640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:46.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.600+0000 7f608f7fe640 1 -- 172.21.15.18:0/883275139 <== mon.1 v1:172.21.15.70:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6080001900 con 0x7f609014f6b0 2024-03-29T11:13:46.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.600+0000 7f608f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:46.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.600+0000 7f6097227640 10 monclient: get_monmap_and_config success 2024-03-29T11:13:46.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.600+0000 7f6097227640 4 set_mon_vals no callback set 2024-03-29T11:13:46.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.600+0000 7f608f7fe640 1 -- 172.21.15.18:0/883275139 <== mon.1 v1:172.21.15.70:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f6080001cd0 con 0x7f609014f6b0 2024-03-29T11:13:46.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.600+0000 7f608f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:46.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.600+0000 7f608f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:13:46.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.600+0000 7f608f7fe640 10 monclient: dump: 2024-03-29T11:13:46.584 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:13:46.584 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:46.584 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:46.584 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:46.584 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:46.585 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:46.585 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:13:46.585 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:13:46.585 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:13:46.585 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:46.585 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.601+0000 7f6097227640 10 monclient: shutdown 2024-03-29T11:13:46.585 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.601+0000 7f6097227640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:46.585 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.601+0000 7f6097227640 1 -- 172.21.15.18:0/883275139 >> v1:172.21.15.70:6789/0 conn(0x7f609014f6b0 legacy=0x7f609014fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:46.586 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.605+0000 7f6097227640 1 -- 172.21.15.18:0/883275139 shutdown_connections 2024-03-29T11:13:46.586 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.605+0000 7f6097227640 1 -- 172.21.15.18:0/883275139 >> 172.21.15.18:0/883275139 conn(0x7f60900b35e0 msgr2=0x7f60900b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:46.586 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.606+0000 7f6097227640 1 -- 172.21.15.18:0/883275139 shutdown_connections 2024-03-29T11:13:46.586 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.606+0000 7f6097227640 1 -- 172.21.15.18:0/883275139 wait complete. 2024-03-29T11:13:46.586 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.606+0000 7f6097227640 10 monclient: build_initial_monmap 2024-03-29T11:13:46.586 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.606+0000 7f6097227640 10 monclient: monmap: 2024-03-29T11:13:46.586 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:13:46.586 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:13:46.586 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:13:46.587 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:13:46.587 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:46.587 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:46.587 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:13:46.587 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:13:46.587 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:13:46.587 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:46.587 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.607+0000 7f6097227640 1 librados: starting msgr at 2024-03-29T11:13:46.587 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.607+0000 7f6097227640 1 librados: starting objecter 2024-03-29T11:13:46.587 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.607+0000 7f6097227640 1 Processor -- start 2024-03-29T11:13:46.588 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.607+0000 7f6097227640 1 -- start start 2024-03-29T11:13:46.588 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.607+0000 7f6097227640 1 librados: setting wanted keys 2024-03-29T11:13:46.588 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.607+0000 7f6097227640 1 librados: calling monclient init 2024-03-29T11:13:46.588 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.607+0000 7f6097227640 10 monclient: init 2024-03-29T11:13:46.588 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.608+0000 7f6097227640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:46.588 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.608+0000 7f6097227640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:13:46.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.612+0000 7f6097227640 10 monclient(hunting): picked mon.noname-a con 0x7f60900af6c0 addr v1:172.21.15.18:6789/0 2024-03-29T11:13:46.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.614+0000 7f6094fc3640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f60900af6c0 0x7f60901f0130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:59141/0 (socket says 172.21.15.18:59141) 2024-03-29T11:13:46.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.615+0000 7f608ffff640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f609014f6b0 0x7f6090084e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:46507/0 (socket says 172.21.15.18:46507) 2024-03-29T11:13:46.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.615+0000 7f6097227640 10 monclient(hunting): picked mon.noname-c con 0x7f609014f6b0 addr v1:172.21.15.18:6790/0 2024-03-29T11:13:46.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.615+0000 7f6094fc3640 1 -- 172.21.15.18:0/933950387 learned_addr learned my addr 172.21.15.18:0/933950387 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:13:46.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.615+0000 7f6097227640 10 monclient(hunting): picked mon.noname-b con 0x7f6090152980 addr v1:172.21.15.70:6789/0 2024-03-29T11:13:46.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.615+0000 7f6097227640 1 -- 172.21.15.18:0/933950387 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6090086a30 con 0x7f60900af6c0 2024-03-29T11:13:46.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.615+0000 7f6097227640 1 -- 172.21.15.18:0/933950387 --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60900843d0 con 0x7f6090152980 2024-03-29T11:13:46.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.615+0000 7f6097227640 1 -- 172.21.15.18:0/933950387 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60901efc10 con 0x7f609014f6b0 2024-03-29T11:13:46.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.615+0000 7f6097227640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:46.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.615+0000 7f6097227640 10 monclient(hunting): authenticate will time out at 1052.888729s 2024-03-29T11:13:46.599 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.615+0000 7f608d7fa640 10 client.?.objecter ms_handle_connect 0x7f60900af6c0 2024-03-29T11:13:46.599 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.618+0000 7f608d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:46.599 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.618+0000 7f608d7fa640 10 client.?.objecter ms_handle_connect 0x7f609014f6b0 2024-03-29T11:13:46.599 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.618+0000 7f608d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:46.599 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.618+0000 7f608d7fa640 1 -- 172.21.15.18:0/933950387 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1805888729 0 0) 0x7f60901efc10 con 0x7f609014f6b0 2024-03-29T11:13:46.599 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.618+0000 7f608d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:13:46.600 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.618+0000 7f608d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:46.600 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.618+0000 7f608d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:46.600 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.618+0000 7f608d7fa640 10 monclient(hunting): my global_id is 4427 2024-03-29T11:13:46.600 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.618+0000 7f608d7fa640 1 -- 172.21.15.18:0/933950387 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f607c004eb0 con 0x7f609014f6b0 2024-03-29T11:13:46.600 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.618+0000 7f608d7fa640 1 -- 172.21.15.18:0/933950387 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2688645624 0 0) 0x7f6090086a30 con 0x7f60900af6c0 2024-03-29T11:13:46.600 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.618+0000 7f608d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:13:46.600 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.618+0000 7f608d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:46.600 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.618+0000 7f608d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:46.600 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.618+0000 7f608d7fa640 10 monclient(hunting): my global_id is 4383 2024-03-29T11:13:46.600 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.618+0000 7f608d7fa640 1 -- 172.21.15.18:0/933950387 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60901efc10 con 0x7f60900af6c0 2024-03-29T11:13:46.600 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.618+0000 7f608d7fa640 1 -- 172.21.15.18:0/933950387 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3603114270 0 0) 0x7f607c004eb0 con 0x7f609014f6b0 2024-03-29T11:13:46.600 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.619+0000 7f608d7fa640 1 -- 172.21.15.18:0/933950387 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6090086a30 con 0x7f609014f6b0 2024-03-29T11:13:46.600 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.619+0000 7f608d7fa640 1 -- 172.21.15.18:0/933950387 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f6080001cd0 con 0x7f609014f6b0 2024-03-29T11:13:46.600 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.619+0000 7f608d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:46.600 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.619+0000 7f608d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:13:46.601 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.619+0000 7f608d7fa640 10 monclient(hunting): dump: 2024-03-29T11:13:46.601 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:13:46.601 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:46.601 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:46.601 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:46.601 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:46.601 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:46.601 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:13:46.601 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:13:46.601 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:13:46.601 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:46.601 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.619+0000 7f608d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:13:46.601 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.619+0000 7f608d7fa640 1 -- 172.21.15.18:0/933950387 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3595461762 0 0) 0x7f6090086a30 con 0x7f609014f6b0 2024-03-29T11:13:46.601 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.619+0000 7f608d7fa640 1 -- 172.21.15.18:0/933950387 >> v1:172.21.15.70:6789/0 conn(0x7f6090152980 legacy=0x7f609014e6a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:13:46.601 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.620+0000 7f608d7fa640 1 -- 172.21.15.18:0/933950387 >> v1:172.21.15.18:6789/0 conn(0x7f60900af6c0 legacy=0x7f60901f0130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:46.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.620+0000 7f608d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:13:46.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.620+0000 7f608d7fa640 1 monclient: found mon.c 2024-03-29T11:13:46.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.620+0000 7f608d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:46.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.620+0000 7f608d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:13:46.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.620+0000 7f608d7fa640 1 -- 172.21.15.18:0/933950387 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60901f0840 con 0x7f609014f6b0 2024-03-29T11:13:46.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.620+0000 7f608d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:13:46.602 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.620+0000 7f608d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:13:46.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.620+0000 7f608d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:46.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.620+0000 7f6097227640 5 monclient: authenticate success, global_id 4427 2024-03-29T11:13:46.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.620+0000 7f6097227640 10 monclient: _renew_subs 2024-03-29T11:13:46.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.620+0000 7f6097227640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:13:46.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.620+0000 7f6097227640 1 -- 172.21.15.18:0/933950387 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6090070b00 con 0x7f609014f6b0 2024-03-29T11:13:46.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.620+0000 7f6097227640 10 client.4427.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:13:46.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.620+0000 7f6097227640 10 monclient: _renew_subs 2024-03-29T11:13:46.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.620+0000 7f608d7fa640 1 -- 172.21.15.18:0/933950387 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60800039a0 con 0x7f609014f6b0 2024-03-29T11:13:46.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.620+0000 7f6097227640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:13:46.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.620+0000 7f6097227640 1 -- 172.21.15.18:0/933950387 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6090085730 con 0x7f609014f6b0 2024-03-29T11:13:46.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.620+0000 7f6097227640 1 librados: init done 2024-03-29T11:13:46.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.620+0000 7f608d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:46.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.620+0000 7f608d7fa640 1 -- 172.21.15.18:0/933950387 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f6080003df0 con 0x7f609014f6b0 2024-03-29T11:13:46.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.620+0000 7f608d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:46.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.620+0000 7f608d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:13:46.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.620+0000 7f608d7fa640 10 monclient: dump: 2024-03-29T11:13:46.604 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:13:46.604 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:46.604 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:46.604 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:46.604 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:46.604 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:46.604 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:13:46.605 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:13:46.605 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:13:46.605 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:46.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.621+0000 7f608effd640 4 set_mon_vals no callback set 2024-03-29T11:13:46.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.622+0000 7f608d7fa640 1 -- 172.21.15.18:0/933950387 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f6080003f50 con 0x7f609014f6b0 2024-03-29T11:13:46.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.623+0000 7f608d7fa640 1 -- 172.21.15.18:0/933950387 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f6080084aa0 con 0x7f609014f6b0 2024-03-29T11:13:46.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.623+0000 7f608d7fa640 10 client.4427.objecter ms_dispatch 0x7f6090000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:13:46.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.623+0000 7f608d7fa640 3 client.4427.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:13:46.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.623+0000 7f608d7fa640 3 client.4427.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:13:46.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.623+0000 7f608d7fa640 20 client.4427.objecter dump_active .. 0 homeless 2024-03-29T11:13:46.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.623+0000 7f6097227640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:46.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.623+0000 7f6097227640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:46.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.623+0000 7f6097227640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:13:46.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.623+0000 7f6097227640 1 -- 172.21.15.18:0/933950387 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6070005080 con 0x7f609014f6b0 2024-03-29T11:13:46.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.624+0000 7f6094fc3640 10 monclient: get_auth_request con 0x7f607c06db80 auth_method 0 2024-03-29T11:13:46.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.626+0000 7f608d7fa640 10 client.4427.objecter ms_handle_connect 0x7f607c06db80 2024-03-29T11:13:46.613 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.632+0000 7f608d7fa640 1 -- 172.21.15.18:0/933950387 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6080050380 con 0x7f609014f6b0 2024-03-29T11:13:46.613 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.632+0000 7f608d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:46.613 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.632+0000 7f608d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:13:46.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.954+0000 7f6097227640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:46.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.954+0000 7f6097227640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:46.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.954+0000 7f6097227640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:13:46.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.954+0000 7f6097227640 1 -- 172.21.15.18:0/933950387 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6070005330 con 0x7f609014f6b0 2024-03-29T11:13:46.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.955+0000 7f608d7fa640 1 -- 172.21.15.18:0/933950387 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f60800522b0 con 0x7f609014f6b0 2024-03-29T11:13:46.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.955+0000 7f608d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:46.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.955+0000 7f608d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:13:46.935 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:13:46.935 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:13:46.940 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.959+0000 7f6097227640 10 librados: watch_flush enter 2024-03-29T11:13:46.940 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.959+0000 7f6097227640 10 librados: watch_flush exit 2024-03-29T11:13:46.940 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.959+0000 7f6097227640 20 client.4427.objecter shutdown clearing up homeless session... 2024-03-29T11:13:46.940 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.959+0000 7f6097227640 10 client.4427.objecter successfully canceled tick 2024-03-29T11:13:46.940 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.959+0000 7f6097227640 1 -- 172.21.15.18:0/933950387 >> v1:172.21.15.70:6816/3101871562 conn(0x7f607c06db80 legacy=0x7f607c070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:46.940 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.959+0000 7f6097227640 10 monclient: shutdown 2024-03-29T11:13:46.940 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.959+0000 7f6097227640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:46.940 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.959+0000 7f6097227640 1 -- 172.21.15.18:0/933950387 >> v1:172.21.15.18:6790/0 conn(0x7f609014f6b0 legacy=0x7f6090084e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:46.940 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.960+0000 7f6097227640 1 -- 172.21.15.18:0/933950387 shutdown_connections 2024-03-29T11:13:46.940 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.960+0000 7f6097227640 1 -- 172.21.15.18:0/933950387 >> 172.21.15.18:0/933950387 conn(0x7f60900b35e0 msgr2=0x7f6090151aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:46.942 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.963+0000 7f6097227640 1 -- 172.21.15.18:0/933950387 shutdown_connections 2024-03-29T11:13:46.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.964+0000 7f6097227640 1 -- 172.21.15.18:0/933950387 wait complete. 2024-03-29T11:13:46.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:46.965+0000 7f6097227640 1 librados: shutdown 2024-03-29T11:13:46.960 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:13:53.766 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:13:53.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.922+0000 7f1042c55640 10 monclient: get_monmap_and_config 2024-03-29T11:13:53.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.922+0000 7f1042c55640 10 monclient: build_initial_monmap 2024-03-29T11:13:53.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.922+0000 7f1042c55640 10 monclient: monmap: 2024-03-29T11:13:53.901 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:13:53.902 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:53.902 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:13:53.924483+0000 2024-03-29T11:13:53.902 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:13:53.924483+0000 2024-03-29T11:13:53.902 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:53.902 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:53.902 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:13:53.902 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:13:53.902 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:13:53.902 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:53.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.923+0000 7f1042c55640 1 Processor -- start 2024-03-29T11:13:53.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.923+0000 7f1042c55640 1 -- start start 2024-03-29T11:13:53.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.923+0000 7f1042c55640 10 monclient: init 2024-03-29T11:13:53.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.924+0000 7f1042c55640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:53.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.924+0000 7f1042c55640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:13:53.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.924+0000 7f1042c55640 10 monclient(hunting): picked mon.noname-c con 0x7f103c1457f0 addr v1:172.21.15.70:6789/0 2024-03-29T11:13:53.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.924+0000 7f1042c55640 10 monclient(hunting): picked mon.noname-b con 0x7f103c13f650 addr v1:172.21.15.18:6790/0 2024-03-29T11:13:53.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.924+0000 7f1042c55640 10 monclient(hunting): picked mon.noname-a con 0x7f103c0a9d70 addr v1:172.21.15.18:6789/0 2024-03-29T11:13:53.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.924+0000 7f1042c55640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f103c0aaef0 con 0x7f103c0a9d70 2024-03-29T11:13:53.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.924+0000 7f1042c55640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f103c145bd0 con 0x7f103c1457f0 2024-03-29T11:13:53.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.924+0000 7f1042c55640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f103c145eb0 con 0x7f103c13f650 2024-03-29T11:13:53.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.924+0000 7f1042c55640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:53.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.924+0000 7f1042c55640 10 monclient(hunting): authenticate will time out at 1060.197401s 2024-03-29T11:13:53.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.924+0000 7f103bfff640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f103c13f650 0x7f103c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:36112/0 (socket says 172.21.15.18:36112) 2024-03-29T11:13:53.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.924+0000 7f10409f1640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f103c1457f0 0x7f103c13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.18:40696/0 (socket says 172.21.15.18:40696) 2024-03-29T11:13:53.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.924+0000 7f10409f1640 1 -- 172.21.15.18:0/3880335687 learned_addr learned my addr 172.21.15.18:0/3880335687 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:13:53.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.925+0000 7f103b7fe640 1 -- 172.21.15.18:0/3880335687 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2016229607 0 0) 0x7f103c145eb0 con 0x7f103c13f650 2024-03-29T11:13:53.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.925+0000 7f103b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:13:53.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.925+0000 7f103b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:53.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.925+0000 7f103b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:53.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.925+0000 7f103b7fe640 10 monclient(hunting): my global_id is 4430 2024-03-29T11:13:53.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.925+0000 7f103b7fe640 1 -- 172.21.15.18:0/3880335687 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f102c003520 con 0x7f103c13f650 2024-03-29T11:13:53.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.925+0000 7f103b7fe640 1 -- 172.21.15.18:0/3880335687 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1415808160 0 0) 0x7f103c145bd0 con 0x7f103c1457f0 2024-03-29T11:13:53.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.925+0000 7f103b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:13:53.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.925+0000 7f103b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:53.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.925+0000 7f103b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:53.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.925+0000 7f103b7fe640 10 monclient(hunting): my global_id is 4372 2024-03-29T11:13:53.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.925+0000 7f103b7fe640 1 -- 172.21.15.18:0/3880335687 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f103c145eb0 con 0x7f103c1457f0 2024-03-29T11:13:53.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.925+0000 7f103b7fe640 1 -- 172.21.15.18:0/3880335687 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3887690461 0 0) 0x7f102c003520 con 0x7f103c13f650 2024-03-29T11:13:53.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.926+0000 7f103b7fe640 1 -- 172.21.15.18:0/3880335687 >> v1:172.21.15.70:6789/0 conn(0x7f103c1457f0 legacy=0x7f103c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:53.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.926+0000 7f103b7fe640 1 -- 172.21.15.18:0/3880335687 >> v1:172.21.15.18:6789/0 conn(0x7f103c0a9d70 legacy=0x7f103c14d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:13:53.909 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.927+0000 7f103b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:13:53.909 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.927+0000 7f103b7fe640 1 monclient: found mon.noname-b 2024-03-29T11:13:53.909 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.927+0000 7f103b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:53.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.927+0000 7f103b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:13:53.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.927+0000 7f103b7fe640 1 -- 172.21.15.18:0/3880335687 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f103c146340 con 0x7f103c13f650 2024-03-29T11:13:53.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.927+0000 7f103b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:13:53.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.927+0000 7f103b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:13:53.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.927+0000 7f103b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:53.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.927+0000 7f1042c55640 5 monclient: authenticate success, global_id 4430 2024-03-29T11:13:53.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.927+0000 7f1042c55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:53.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.927+0000 7f103b7fe640 1 -- 172.21.15.18:0/3880335687 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f1028001690 con 0x7f103c13f650 2024-03-29T11:13:53.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.927+0000 7f103b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:53.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.927+0000 7f103b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:13:53.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.927+0000 7f103b7fe640 10 monclient: dump: 2024-03-29T11:13:53.911 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:13:53.911 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:53.911 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:53.911 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:53.911 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:53.911 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:53.911 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:13:53.911 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:13:53.911 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:13:53.911 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:53.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.927+0000 7f1042c55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:13:53.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.928+0000 7f103b7fe640 1 -- 172.21.15.18:0/3880335687 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10280018b0 con 0x7f103c13f650 2024-03-29T11:13:53.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.928+0000 7f103b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:53.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.928+0000 7f1042c55640 10 monclient: get_monmap_and_config success 2024-03-29T11:13:53.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.928+0000 7f1042c55640 4 set_mon_vals no callback set 2024-03-29T11:13:53.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.928+0000 7f1042c55640 10 monclient: shutdown 2024-03-29T11:13:53.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.928+0000 7f1042c55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:53.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.928+0000 7f1042c55640 1 -- 172.21.15.18:0/3880335687 >> v1:172.21.15.18:6790/0 conn(0x7f103c13f650 legacy=0x7f103c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:53.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.928+0000 7f1042c55640 1 -- 172.21.15.18:0/3880335687 shutdown_connections 2024-03-29T11:13:53.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.928+0000 7f1042c55640 1 -- 172.21.15.18:0/3880335687 >> 172.21.15.18:0/3880335687 conn(0x7f103c066a00 msgr2=0x7f103c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:53.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.928+0000 7f1042c55640 1 -- 172.21.15.18:0/3880335687 shutdown_connections 2024-03-29T11:13:53.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.929+0000 7f1042c55640 1 -- 172.21.15.18:0/3880335687 wait complete. 2024-03-29T11:13:53.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.929+0000 7f1042c55640 10 monclient: build_initial_monmap 2024-03-29T11:13:53.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.929+0000 7f1042c55640 10 monclient: monmap: 2024-03-29T11:13:53.912 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:13:53.912 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:13:53.912 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:13:53.912 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:13:53.912 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:13:53.912 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:53.912 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:13:53.913 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:13:53.913 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:13:53.913 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:53.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.930+0000 7f1042c55640 1 librados: starting msgr at 2024-03-29T11:13:53.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.930+0000 7f1042c55640 1 librados: starting objecter 2024-03-29T11:13:53.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.930+0000 7f1042c55640 1 Processor -- start 2024-03-29T11:13:53.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.930+0000 7f1042c55640 1 -- start start 2024-03-29T11:13:53.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.930+0000 7f1042c55640 1 librados: setting wanted keys 2024-03-29T11:13:53.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.930+0000 7f1042c55640 1 librados: calling monclient init 2024-03-29T11:13:53.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.930+0000 7f1042c55640 10 monclient: init 2024-03-29T11:13:53.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.931+0000 7f1042c55640 10 monclient: _reopen_session rank -1 2024-03-29T11:13:53.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.931+0000 7f1042c55640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:13:53.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.931+0000 7f1042c55640 10 monclient(hunting): picked mon.noname-b con 0x7f103c0a9d70 addr v1:172.21.15.70:6789/0 2024-03-29T11:13:53.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.931+0000 7f1042c55640 10 monclient(hunting): picked mon.noname-c con 0x7f103c13f650 addr v1:172.21.15.18:6790/0 2024-03-29T11:13:53.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.931+0000 7f1042c55640 10 monclient(hunting): picked mon.noname-a con 0x7f103c1457f0 addr v1:172.21.15.18:6789/0 2024-03-29T11:13:53.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.931+0000 7f1042c55640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f103c142e40 con 0x7f103c1457f0 2024-03-29T11:13:53.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.931+0000 7f1042c55640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f103c141b50 con 0x7f103c0a9d70 2024-03-29T11:13:53.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.931+0000 7f1042c55640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f103c141e30 con 0x7f103c13f650 2024-03-29T11:13:53.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.931+0000 7f1042c55640 10 monclient(hunting): _renew_subs 2024-03-29T11:13:53.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.931+0000 7f1042c55640 10 monclient(hunting): authenticate will time out at 1060.204471s 2024-03-29T11:13:53.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.932+0000 7f103bfff640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f103c13f650 0x7f103c135040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:38222/0 (socket says 172.21.15.18:38222) 2024-03-29T11:13:53.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.932+0000 7f103bfff640 1 -- 172.21.15.18:0/2004153597 learned_addr learned my addr 172.21.15.18:0/2004153597 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:13:53.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.932+0000 7f10397fa640 10 client.?.objecter ms_handle_connect 0x7f103c13f650 2024-03-29T11:13:53.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.932+0000 7f10397fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:13:53.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.932+0000 7f10397fa640 1 -- 172.21.15.18:0/2004153597 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 825286838 0 0) 0x7f103c141e30 con 0x7f103c13f650 2024-03-29T11:13:53.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.932+0000 7f10397fa640 10 monclient(hunting): _negotiate 2024-03-29T11:13:53.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.932+0000 7f10397fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:13:53.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.932+0000 7f10397fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:13:53.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.932+0000 7f10397fa640 10 monclient(hunting): my global_id is 4433 2024-03-29T11:13:53.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.932+0000 7f10397fa640 1 -- 172.21.15.18:0/2004153597 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1020004eb0 con 0x7f103c13f650 2024-03-29T11:13:53.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.932+0000 7f10397fa640 1 -- 172.21.15.18:0/2004153597 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 580963309 0 0) 0x7f1020004eb0 con 0x7f103c13f650 2024-03-29T11:13:53.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.932+0000 7f10397fa640 1 -- 172.21.15.18:0/2004153597 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f103c141e30 con 0x7f103c13f650 2024-03-29T11:13:53.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.932+0000 7f10397fa640 1 -- 172.21.15.18:0/2004153597 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f10280016d0 con 0x7f103c13f650 2024-03-29T11:13:53.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.932+0000 7f10397fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:53.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.932+0000 7f10397fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:13:53.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.932+0000 7f10397fa640 10 monclient(hunting): dump: 2024-03-29T11:13:53.921 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:13:53.921 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:53.921 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:53.921 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:53.921 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:53.921 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:53.922 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:13:53.922 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:13:53.922 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:13:53.922 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:53.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.932+0000 7f10397fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:13:53.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.932+0000 7f10397fa640 1 -- 172.21.15.18:0/2004153597 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2872053292 0 0) 0x7f103c141e30 con 0x7f103c13f650 2024-03-29T11:13:53.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.932+0000 7f10397fa640 1 -- 172.21.15.18:0/2004153597 >> v1:172.21.15.70:6789/0 conn(0x7f103c0a9d70 legacy=0x7f103c142730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:53.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.932+0000 7f10397fa640 1 -- 172.21.15.18:0/2004153597 >> v1:172.21.15.18:6789/0 conn(0x7f103c1457f0 legacy=0x7f103c1e2b60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:13:53.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.933+0000 7f10397fa640 10 monclient: _finish_hunting 0 2024-03-29T11:13:53.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.933+0000 7f10397fa640 1 monclient: found mon.c 2024-03-29T11:13:53.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.933+0000 7f10397fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:13:53.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.933+0000 7f10397fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:13:53.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.933+0000 7f10397fa640 1 -- 172.21.15.18:0/2004153597 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f103c1400d0 con 0x7f103c13f650 2024-03-29T11:13:53.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.933+0000 7f10397fa640 10 monclient: _finish_auth 0 2024-03-29T11:13:53.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.933+0000 7f10397fa640 10 monclient: _check_auth_tickets 2024-03-29T11:13:53.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.933+0000 7f10397fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:13:53.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.933+0000 7f1042c55640 5 monclient: authenticate success, global_id 4433 2024-03-29T11:13:53.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.933+0000 7f1042c55640 10 monclient: _renew_subs 2024-03-29T11:13:53.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.933+0000 7f1042c55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:13:53.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.933+0000 7f1042c55640 1 -- 172.21.15.18:0/2004153597 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f103c13fda0 con 0x7f103c13f650 2024-03-29T11:13:53.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.933+0000 7f1042c55640 10 client.4433.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:13:53.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.933+0000 7f1042c55640 10 monclient: _renew_subs 2024-03-29T11:13:53.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.933+0000 7f1042c55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:13:53.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.933+0000 7f1042c55640 1 -- 172.21.15.18:0/2004153597 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f103c137d50 con 0x7f103c13f650 2024-03-29T11:13:53.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.933+0000 7f1042c55640 1 librados: init done 2024-03-29T11:13:53.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.934+0000 7f10397fa640 1 -- 172.21.15.18:0/2004153597 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1028001210 con 0x7f103c13f650 2024-03-29T11:13:53.924 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.934+0000 7f10397fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:13:53.924 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.934+0000 7f103a7fc640 4 set_mon_vals no callback set 2024-03-29T11:13:53.924 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.934+0000 7f10397fa640 1 -- 172.21.15.18:0/2004153597 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f1028004df0 con 0x7f103c13f650 2024-03-29T11:13:53.924 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.934+0000 7f10397fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:13:53.924 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.934+0000 7f10397fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:13:53.924 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.934+0000 7f10397fa640 10 monclient: dump: 2024-03-29T11:13:53.924 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:13:53.924 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:13:53.924 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:53.924 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:13:53.924 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:13:53.924 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:13:53.924 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:13:53.925 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:13:53.925 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:13:53.925 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:13:53.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.935+0000 7f1042c55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:53.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.935+0000 7f1042c55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:53.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.935+0000 7f1042c55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:13:53.926 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.935+0000 7f1042c55640 1 -- 172.21.15.18:0/2004153597 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f101c005080 con 0x7f103c13f650 2024-03-29T11:13:53.926 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.936+0000 7f10397fa640 1 -- 172.21.15.18:0/2004153597 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f1028004f50 con 0x7f103c13f650 2024-03-29T11:13:53.926 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.936+0000 7f10397fa640 1 -- 172.21.15.18:0/2004153597 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f1028084dd0 con 0x7f103c13f650 2024-03-29T11:13:53.926 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.936+0000 7f10397fa640 10 client.4433.objecter ms_dispatch 0x7f103c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:13:53.926 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.936+0000 7f10397fa640 3 client.4433.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:13:53.926 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.936+0000 7f10397fa640 3 client.4433.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:13:53.926 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.936+0000 7f10397fa640 20 client.4433.objecter dump_active .. 0 homeless 2024-03-29T11:13:53.926 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.936+0000 7f10409f1640 10 monclient: get_auth_request con 0x7f102006dbf0 auth_method 0 2024-03-29T11:13:53.926 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.937+0000 7f10397fa640 10 client.4433.objecter ms_handle_connect 0x7f102006dbf0 2024-03-29T11:13:53.927 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.944+0000 7f10397fa640 1 -- 172.21.15.18:0/2004153597 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f10280504e0 con 0x7f103c13f650 2024-03-29T11:13:53.927 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.944+0000 7f10397fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:13:53.927 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:53.944+0000 7f10397fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:13:54.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:54.267+0000 7f1042c55640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:54.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:54.267+0000 7f1042c55640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:54.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:54.267+0000 7f1042c55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:13:54.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:54.267+0000 7f1042c55640 1 -- 172.21.15.18:0/2004153597 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f101c005330 con 0x7f103c13f650 2024-03-29T11:13:54.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:54.268+0000 7f10397fa640 1 -- 172.21.15.18:0/2004153597 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f10280525e0 con 0x7f103c13f650 2024-03-29T11:13:54.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:54.268+0000 7f10397fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:13:54.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:54.268+0000 7f10397fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:13:54.247 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:13:54.247 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:13:54.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:54.271+0000 7f1042c55640 10 librados: watch_flush enter 2024-03-29T11:13:54.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:54.271+0000 7f1042c55640 10 librados: watch_flush exit 2024-03-29T11:13:54.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:54.271+0000 7f1042c55640 20 client.4433.objecter shutdown clearing up homeless session... 2024-03-29T11:13:54.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:54.271+0000 7f1042c55640 10 client.4433.objecter successfully canceled tick 2024-03-29T11:13:54.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:54.271+0000 7f1042c55640 1 -- 172.21.15.18:0/2004153597 >> v1:172.21.15.70:6816/3101871562 conn(0x7f102006dbf0 legacy=0x7f10200700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:54.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:54.271+0000 7f1042c55640 10 monclient: shutdown 2024-03-29T11:13:54.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:54.271+0000 7f1042c55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:13:54.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:54.271+0000 7f1042c55640 1 -- 172.21.15.18:0/2004153597 >> v1:172.21.15.18:6790/0 conn(0x7f103c13f650 legacy=0x7f103c135040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:13:54.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:54.271+0000 7f1042c55640 1 -- 172.21.15.18:0/2004153597 shutdown_connections 2024-03-29T11:13:54.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:54.271+0000 7f1042c55640 1 -- 172.21.15.18:0/2004153597 >> 172.21.15.18:0/2004153597 conn(0x7f103c066a00 msgr2=0x7f103c13b3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:13:54.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:54.271+0000 7f1042c55640 1 -- 172.21.15.18:0/2004153597 shutdown_connections 2024-03-29T11:13:54.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:54.271+0000 7f1042c55640 1 -- 172.21.15.18:0/2004153597 wait complete. 2024-03-29T11:13:54.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:13:54.271+0000 7f1042c55640 1 librados: shutdown 2024-03-29T11:13:54.264 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:01.070 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:01.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.228+0000 7fac886ff640 10 monclient: get_monmap_and_config 2024-03-29T11:14:01.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.228+0000 7fac886ff640 10 monclient: build_initial_monmap 2024-03-29T11:14:01.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.228+0000 7fac886ff640 10 monclient: monmap: 2024-03-29T11:14:01.209 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:14:01.209 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:01.209 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:14:01.228948+0000 2024-03-29T11:14:01.209 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:14:01.228948+0000 2024-03-29T11:14:01.210 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:01.210 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:01.210 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:14:01.210 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:14:01.210 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:14:01.210 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:01.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.228+0000 7fac886ff640 1 Processor -- start 2024-03-29T11:14:01.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.228+0000 7fac886ff640 1 -- start start 2024-03-29T11:14:01.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.228+0000 7fac886ff640 10 monclient: init 2024-03-29T11:14:01.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.229+0000 7fac886ff640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:01.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.229+0000 7fac886ff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:14:01.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.229+0000 7fac886ff640 10 monclient(hunting): picked mon.noname-c con 0x7fac800ae250 addr v1:172.21.15.70:6789/0 2024-03-29T11:14:01.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.229+0000 7fac886ff640 10 monclient(hunting): picked mon.noname-a con 0x7fac8013f970 addr v1:172.21.15.18:6789/0 2024-03-29T11:14:01.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.229+0000 7fac886ff640 10 monclient(hunting): picked mon.noname-b con 0x7fac80142800 addr v1:172.21.15.18:6790/0 2024-03-29T11:14:01.212 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.229+0000 7fac886ff640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac800afe00 con 0x7fac8013f970 2024-03-29T11:14:01.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.229+0000 7fac886ff640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac80142c10 con 0x7fac800ae250 2024-03-29T11:14:01.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.229+0000 7fac886ff640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac800afb20 con 0x7fac80142800 2024-03-29T11:14:01.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.229+0000 7fac886ff640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:01.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.229+0000 7fac886ff640 10 monclient(hunting): authenticate will time out at 1067.501679s 2024-03-29T11:14:01.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.229+0000 7fac86c9c640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fac80142800 0x7fac8014da10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:60464/0 (socket says 172.21.15.18:60464) 2024-03-29T11:14:01.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.229+0000 7fac86c9c640 1 -- 172.21.15.18:0/1767638575 learned_addr learned my addr 172.21.15.18:0/1767638575 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:14:01.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.230+0000 7fac85499640 1 -- 172.21.15.18:0/1767638575 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 370940962 0 0) 0x7fac800afb20 con 0x7fac80142800 2024-03-29T11:14:01.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.230+0000 7fac85499640 10 monclient(hunting): _negotiate 2024-03-29T11:14:01.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.230+0000 7fac85499640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:01.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.230+0000 7fac85499640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:01.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.230+0000 7fac85499640 10 monclient(hunting): my global_id is 4436 2024-03-29T11:14:01.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.230+0000 7fac85499640 1 -- 172.21.15.18:0/1767638575 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac6c003520 con 0x7fac80142800 2024-03-29T11:14:01.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.230+0000 7fac85499640 1 -- 172.21.15.18:0/1767638575 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3664994239 0 0) 0x7fac6c003520 con 0x7fac80142800 2024-03-29T11:14:01.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.230+0000 7fac85499640 1 -- 172.21.15.18:0/1767638575 >> v1:172.21.15.70:6789/0 conn(0x7fac800ae250 legacy=0x7fac8013f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:14:01.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.230+0000 7fac85499640 1 -- 172.21.15.18:0/1767638575 >> v1:172.21.15.18:6789/0 conn(0x7fac8013f970 legacy=0x7fac801420f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:14:01.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.232+0000 7fac85499640 10 monclient: _finish_hunting 0 2024-03-29T11:14:01.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.232+0000 7fac85499640 1 monclient: found mon.noname-b 2024-03-29T11:14:01.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.232+0000 7fac85499640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:01.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.232+0000 7fac85499640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:14:01.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.232+0000 7fac85499640 1 -- 172.21.15.18:0/1767638575 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac800aab80 con 0x7fac80142800 2024-03-29T11:14:01.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.232+0000 7fac85499640 10 monclient: _finish_auth 0 2024-03-29T11:14:01.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.232+0000 7fac85499640 10 monclient: _check_auth_tickets 2024-03-29T11:14:01.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.232+0000 7fac85499640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:01.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.232+0000 7fac85499640 1 -- 172.21.15.18:0/1767638575 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fac7c001690 con 0x7fac80142800 2024-03-29T11:14:01.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.232+0000 7fac85499640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:01.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.232+0000 7fac85499640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:01.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.232+0000 7fac85499640 10 monclient: dump: 2024-03-29T11:14:01.224 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:14:01.224 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:01.224 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:01.224 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:01.224 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:01.224 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:01.224 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:14:01.224 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:14:01.224 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:14:01.224 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:01.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.232+0000 7fac886ff640 5 monclient: authenticate success, global_id 4436 2024-03-29T11:14:01.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.232+0000 7fac886ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:01.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.233+0000 7fac85499640 1 -- 172.21.15.18:0/1767638575 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac7c001860 con 0x7fac80142800 2024-03-29T11:14:01.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.233+0000 7fac85499640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:01.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.233+0000 7fac85499640 1 -- 172.21.15.18:0/1767638575 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fac7c001cb0 con 0x7fac80142800 2024-03-29T11:14:01.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.233+0000 7fac85499640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:01.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.233+0000 7fac85499640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:01.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.233+0000 7fac85499640 10 monclient: dump: 2024-03-29T11:14:01.225 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:14:01.225 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:01.225 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:01.225 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:01.225 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:01.225 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:01.225 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:14:01.225 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:14:01.225 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:14:01.226 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:01.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.233+0000 7fac886ff640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:01.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.233+0000 7fac886ff640 4 set_mon_vals no callback set 2024-03-29T11:14:01.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.234+0000 7fac886ff640 10 monclient: shutdown 2024-03-29T11:14:01.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.234+0000 7fac886ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:01.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.234+0000 7fac886ff640 1 -- 172.21.15.18:0/1767638575 >> v1:172.21.15.18:6790/0 conn(0x7fac80142800 legacy=0x7fac8014da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:01.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.234+0000 7fac886ff640 1 -- 172.21.15.18:0/1767638575 shutdown_connections 2024-03-29T11:14:01.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.234+0000 7fac886ff640 1 -- 172.21.15.18:0/1767638575 >> 172.21.15.18:0/1767638575 conn(0x7fac800b35e0 msgr2=0x7fac800b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:01.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.234+0000 7fac886ff640 1 -- 172.21.15.18:0/1767638575 shutdown_connections 2024-03-29T11:14:01.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.234+0000 7fac886ff640 1 -- 172.21.15.18:0/1767638575 wait complete. 2024-03-29T11:14:01.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.235+0000 7fac886ff640 10 monclient: build_initial_monmap 2024-03-29T11:14:01.226 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.235+0000 7fac886ff640 10 monclient: monmap: 2024-03-29T11:14:01.226 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:14:01.226 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:01.227 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:14:01.227 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:14:01.227 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:01.227 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:01.227 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:14:01.227 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:14:01.227 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:14:01.227 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:01.227 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.236+0000 7fac886ff640 1 librados: starting msgr at 2024-03-29T11:14:01.227 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.236+0000 7fac886ff640 1 librados: starting objecter 2024-03-29T11:14:01.227 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.236+0000 7fac886ff640 1 Processor -- start 2024-03-29T11:14:01.227 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.236+0000 7fac886ff640 1 -- start start 2024-03-29T11:14:01.227 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.236+0000 7fac886ff640 1 librados: setting wanted keys 2024-03-29T11:14:01.227 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.236+0000 7fac886ff640 1 librados: calling monclient init 2024-03-29T11:14:01.227 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.236+0000 7fac886ff640 10 monclient: init 2024-03-29T11:14:01.228 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.237+0000 7fac886ff640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:01.228 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.237+0000 7fac886ff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:14:01.228 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.237+0000 7fac886ff640 10 monclient(hunting): picked mon.noname-a con 0x7fac800ae250 addr v1:172.21.15.18:6789/0 2024-03-29T11:14:01.228 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.237+0000 7fac886ff640 10 monclient(hunting): picked mon.noname-b con 0x7fac8013f970 addr v1:172.21.15.70:6789/0 2024-03-29T11:14:01.228 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.237+0000 7fac886ff640 10 monclient(hunting): picked mon.noname-c con 0x7fac80142800 addr v1:172.21.15.18:6790/0 2024-03-29T11:14:01.228 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.237+0000 7fac886ff640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac80144fc0 con 0x7fac800ae250 2024-03-29T11:14:01.228 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.237+0000 7fac886ff640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac80146030 con 0x7fac8013f970 2024-03-29T11:14:01.228 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.237+0000 7fac886ff640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac80146310 con 0x7fac80142800 2024-03-29T11:14:01.228 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.237+0000 7fac886ff640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:01.228 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.237+0000 7fac886ff640 10 monclient(hunting): authenticate will time out at 1067.509081s 2024-03-29T11:14:01.228 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.237+0000 7fac86c9c640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fac80142800 0x7fac801e5fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:33048/0 (socket says 172.21.15.18:33048) 2024-03-29T11:14:01.228 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.237+0000 7fac86c9c640 1 -- 172.21.15.18:0/3169758372 learned_addr learned my addr 172.21.15.18:0/3169758372 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:14:01.228 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.237+0000 7fac72ffd640 10 client.?.objecter ms_handle_connect 0x7fac80142800 2024-03-29T11:14:01.228 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.237+0000 7fac72ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:01.228 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.238+0000 7fac72ffd640 1 -- 172.21.15.18:0/3169758372 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2908861600 0 0) 0x7fac80146310 con 0x7fac80142800 2024-03-29T11:14:01.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.239+0000 7fac72ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:01.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.239+0000 7fac72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:01.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.239+0000 7fac72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:01.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.239+0000 7fac72ffd640 10 monclient(hunting): my global_id is 4439 2024-03-29T11:14:01.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.239+0000 7fac72ffd640 1 -- 172.21.15.18:0/3169758372 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac64004eb0 con 0x7fac80142800 2024-03-29T11:14:01.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.239+0000 7fac72ffd640 10 client.?.objecter ms_handle_connect 0x7fac8013f970 2024-03-29T11:14:01.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.239+0000 7fac72ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:01.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.239+0000 7fac72ffd640 1 -- 172.21.15.18:0/3169758372 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2984461402 0 0) 0x7fac64004eb0 con 0x7fac80142800 2024-03-29T11:14:01.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.239+0000 7fac72ffd640 1 -- 172.21.15.18:0/3169758372 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac80146310 con 0x7fac80142800 2024-03-29T11:14:01.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.239+0000 7fac72ffd640 1 -- 172.21.15.18:0/3169758372 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fac7c001bb0 con 0x7fac80142800 2024-03-29T11:14:01.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.239+0000 7fac72ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:01.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.239+0000 7fac72ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:14:01.229 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.239+0000 7fac72ffd640 10 monclient(hunting): dump: 2024-03-29T11:14:01.229 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:14:01.230 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:01.230 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:01.230 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:01.230 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:01.230 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:01.230 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:14:01.230 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:14:01.230 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:14:01.230 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:01.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.239+0000 7fac72ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:01.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.239+0000 7fac72ffd640 1 -- 172.21.15.18:0/3169758372 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 412473479 0 0) 0x7fac80146030 con 0x7fac8013f970 2024-03-29T11:14:01.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.239+0000 7fac72ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:01.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.239+0000 7fac72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:01.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.239+0000 7fac72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:01.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.239+0000 7fac72ffd640 10 monclient(hunting): my global_id is 4375 2024-03-29T11:14:01.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.239+0000 7fac72ffd640 1 -- 172.21.15.18:0/3169758372 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac64004eb0 con 0x7fac8013f970 2024-03-29T11:14:01.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.240+0000 7fac72ffd640 1 -- 172.21.15.18:0/3169758372 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2308664409 0 0) 0x7fac80146310 con 0x7fac80142800 2024-03-29T11:14:01.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.240+0000 7fac72ffd640 1 -- 172.21.15.18:0/3169758372 >> v1:172.21.15.70:6789/0 conn(0x7fac8013f970 legacy=0x7fac8013aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:01.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.240+0000 7fac72ffd640 1 -- 172.21.15.18:0/3169758372 >> v1:172.21.15.18:6789/0 conn(0x7fac800ae250 legacy=0x7fac80144200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:01.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.240+0000 7fac72ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:14:01.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.240+0000 7fac72ffd640 1 monclient: found mon.c 2024-03-29T11:14:01.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.240+0000 7fac72ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:01.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.240+0000 7fac72ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:14:01.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.240+0000 7fac72ffd640 1 -- 172.21.15.18:0/3169758372 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac801449c0 con 0x7fac80142800 2024-03-29T11:14:01.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.240+0000 7fac72ffd640 10 monclient: _finish_auth 0 2024-03-29T11:14:01.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.240+0000 7fac72ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:14:01.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.240+0000 7fac72ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:01.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.240+0000 7fac886ff640 5 monclient: authenticate success, global_id 4439 2024-03-29T11:14:01.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.240+0000 7fac886ff640 10 monclient: _renew_subs 2024-03-29T11:14:01.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.240+0000 7fac886ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:14:01.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.240+0000 7fac886ff640 1 -- 172.21.15.18:0/3169758372 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac80144c10 con 0x7fac80142800 2024-03-29T11:14:01.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.240+0000 7fac886ff640 10 client.4439.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:01.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.240+0000 7fac886ff640 10 monclient: _renew_subs 2024-03-29T11:14:01.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.240+0000 7fac886ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:14:01.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.240+0000 7fac886ff640 1 -- 172.21.15.18:0/3169758372 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fac8013b3b0 con 0x7fac80142800 2024-03-29T11:14:01.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.240+0000 7fac886ff640 1 librados: init done 2024-03-29T11:14:01.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.240+0000 7fac72ffd640 1 -- 172.21.15.18:0/3169758372 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac7c004a90 con 0x7fac80142800 2024-03-29T11:14:01.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.240+0000 7fac72ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:01.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.240+0000 7fac72ffd640 1 -- 172.21.15.18:0/3169758372 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fac7c004e60 con 0x7fac80142800 2024-03-29T11:14:01.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.240+0000 7fac72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:01.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.240+0000 7fac72ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:01.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.240+0000 7fac72ffd640 10 monclient: dump: 2024-03-29T11:14:01.232 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:14:01.232 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:01.233 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:01.233 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:01.233 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:01.233 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:01.233 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:14:01.233 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:14:01.233 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:14:01.233 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:01.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.240+0000 7fac84c98640 4 set_mon_vals no callback set 2024-03-29T11:14:01.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.242+0000 7fac886ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:01.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.242+0000 7fac886ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:01.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.242+0000 7fac886ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:14:01.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.242+0000 7fac886ff640 1 -- 172.21.15.18:0/3169758372 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac68005080 con 0x7fac80142800 2024-03-29T11:14:01.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.242+0000 7fac72ffd640 1 -- 172.21.15.18:0/3169758372 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7fac7c005040 con 0x7fac80142800 2024-03-29T11:14:01.233 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.243+0000 7fac72ffd640 1 -- 172.21.15.18:0/3169758372 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7fac7c054a00 con 0x7fac80142800 2024-03-29T11:14:01.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.243+0000 7fac72ffd640 10 client.4439.objecter ms_dispatch 0x7fac80000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:01.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.243+0000 7fac72ffd640 3 client.4439.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:01.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.243+0000 7fac72ffd640 3 client.4439.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:01.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.243+0000 7fac72ffd640 20 client.4439.objecter dump_active .. 0 homeless 2024-03-29T11:14:01.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.243+0000 7fac8649b640 10 monclient: get_auth_request con 0x7fac6406dcb0 auth_method 0 2024-03-29T11:14:01.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.244+0000 7fac72ffd640 10 client.4439.objecter ms_handle_connect 0x7fac6406dcb0 2024-03-29T11:14:01.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.251+0000 7fac72ffd640 1 -- 172.21.15.18:0/3169758372 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fac7c050ca0 con 0x7fac80142800 2024-03-29T11:14:01.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.251+0000 7fac72ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:01.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.251+0000 7fac72ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:01.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.560+0000 7fac886ff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:01.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.560+0000 7fac886ff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:01.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.560+0000 7fac886ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:14:01.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.560+0000 7fac886ff640 1 -- 172.21.15.18:0/3169758372 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac68005330 con 0x7fac80142800 2024-03-29T11:14:01.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.561+0000 7fac72ffd640 1 -- 172.21.15.18:0/3169758372 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7fac7c050e40 con 0x7fac80142800 2024-03-29T11:14:01.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.561+0000 7fac72ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:01.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.561+0000 7fac72ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:01.539 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:14:01.539 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:01.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.565+0000 7fac886ff640 10 librados: watch_flush enter 2024-03-29T11:14:01.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.565+0000 7fac886ff640 10 librados: watch_flush exit 2024-03-29T11:14:01.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.565+0000 7fac886ff640 20 client.4439.objecter shutdown clearing up homeless session... 2024-03-29T11:14:01.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.565+0000 7fac886ff640 10 client.4439.objecter successfully canceled tick 2024-03-29T11:14:01.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.565+0000 7fac886ff640 1 -- 172.21.15.18:0/3169758372 >> v1:172.21.15.70:6816/3101871562 conn(0x7fac6406dcb0 legacy=0x7fac64070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:01.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.565+0000 7fac886ff640 10 monclient: shutdown 2024-03-29T11:14:01.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.565+0000 7fac886ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:01.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.565+0000 7fac886ff640 1 -- 172.21.15.18:0/3169758372 >> v1:172.21.15.18:6790/0 conn(0x7fac80142800 legacy=0x7fac801e5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:01.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.565+0000 7fac886ff640 1 -- 172.21.15.18:0/3169758372 shutdown_connections 2024-03-29T11:14:01.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.565+0000 7fac886ff640 1 -- 172.21.15.18:0/3169758372 >> 172.21.15.18:0/3169758372 conn(0x7fac800b35e0 msgr2=0x7fac800b16c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:01.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.565+0000 7fac886ff640 1 -- 172.21.15.18:0/3169758372 shutdown_connections 2024-03-29T11:14:01.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.565+0000 7fac886ff640 1 -- 172.21.15.18:0/3169758372 wait complete. 2024-03-29T11:14:01.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:01.565+0000 7fac886ff640 1 librados: shutdown 2024-03-29T11:14:01.558 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:08.365 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:08.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.542+0000 7f4567b3d640 10 monclient: get_monmap_and_config 2024-03-29T11:14:08.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.542+0000 7f4567b3d640 10 monclient: build_initial_monmap 2024-03-29T11:14:08.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.542+0000 7f4567b3d640 10 monclient: monmap: 2024-03-29T11:14:08.520 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:14:08.520 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:08.520 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:14:08.543238+0000 2024-03-29T11:14:08.520 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:14:08.543238+0000 2024-03-29T11:14:08.520 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:08.520 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:08.520 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:14:08.520 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:14:08.520 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:14:08.520 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:08.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.545+0000 7f4567b3d640 1 Processor -- start 2024-03-29T11:14:08.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.545+0000 7f4567b3d640 1 -- start start 2024-03-29T11:14:08.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.545+0000 7f4567b3d640 10 monclient: init 2024-03-29T11:14:08.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.546+0000 7f4567b3d640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:08.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.546+0000 7f4567b3d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:14:08.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.546+0000 7f4567b3d640 10 monclient(hunting): picked mon.noname-a con 0x7f45601457f0 addr v1:172.21.15.18:6789/0 2024-03-29T11:14:08.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.547+0000 7f4567b3d640 10 monclient(hunting): picked mon.noname-b con 0x7f456013f650 addr v1:172.21.15.18:6790/0 2024-03-29T11:14:08.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.547+0000 7f4567b3d640 10 monclient(hunting): picked mon.noname-c con 0x7f45600a9d70 addr v1:172.21.15.70:6789/0 2024-03-29T11:14:08.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.547+0000 7f4567b3d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45600aaef0 con 0x7f45601457f0 2024-03-29T11:14:08.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.547+0000 7f4567b3d640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4560145bd0 con 0x7f45600a9d70 2024-03-29T11:14:08.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.547+0000 7f4567b3d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4560145eb0 con 0x7f456013f650 2024-03-29T11:14:08.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.547+0000 7f4567b3d640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:08.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.547+0000 7f4567b3d640 10 monclient(hunting): authenticate will time out at 1074.819082s 2024-03-29T11:14:08.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.550+0000 7f45660da640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f45600a9d70 0x7f456014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.18:58589/0 (socket says 172.21.15.18:58589) 2024-03-29T11:14:08.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.552+0000 7f45658d9640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f45601457f0 0x7f456013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:35737/0 (socket says 172.21.15.18:35737) 2024-03-29T11:14:08.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.557+0000 7f45660da640 1 -- 172.21.15.18:0/2650886721 learned_addr learned my addr 172.21.15.18:0/2650886721 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:14:08.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.551+0000 7f45650d8640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f456013f650 0x7f45600b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:59361/0 (socket says 172.21.15.18:59361) 2024-03-29T11:14:08.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.557+0000 7f45648d7640 1 -- 172.21.15.18:0/2650886721 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2465207212 0 0) 0x7f4560145eb0 con 0x7f456013f650 2024-03-29T11:14:08.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.557+0000 7f45648d7640 10 monclient(hunting): _negotiate 2024-03-29T11:14:08.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.557+0000 7f45648d7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:08.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.557+0000 7f45648d7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:08.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.557+0000 7f45648d7640 10 monclient(hunting): my global_id is 4442 2024-03-29T11:14:08.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.557+0000 7f45648d7640 1 -- 172.21.15.18:0/2650886721 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f454c003520 con 0x7f456013f650 2024-03-29T11:14:08.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.557+0000 7f45648d7640 1 -- 172.21.15.18:0/2650886721 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3917488407 0 0) 0x7f45600aaef0 con 0x7f45601457f0 2024-03-29T11:14:08.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.557+0000 7f45648d7640 10 monclient(hunting): _negotiate 2024-03-29T11:14:08.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.557+0000 7f45648d7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:08.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.557+0000 7f45648d7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:08.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.557+0000 7f45648d7640 10 monclient(hunting): my global_id is 4386 2024-03-29T11:14:08.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.557+0000 7f45648d7640 1 -- 172.21.15.18:0/2650886721 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4560145eb0 con 0x7f45601457f0 2024-03-29T11:14:08.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.558+0000 7f45648d7640 1 -- 172.21.15.18:0/2650886721 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1613170789 0 0) 0x7f454c003520 con 0x7f456013f650 2024-03-29T11:14:08.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.558+0000 7f45648d7640 1 -- 172.21.15.18:0/2650886721 >> v1:172.21.15.70:6789/0 conn(0x7f45600a9d70 legacy=0x7f456014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:14:08.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.558+0000 7f45648d7640 1 -- 172.21.15.18:0/2650886721 >> v1:172.21.15.18:6789/0 conn(0x7f45601457f0 legacy=0x7f456013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:08.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.558+0000 7f45648d7640 10 monclient: _finish_hunting 0 2024-03-29T11:14:08.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.558+0000 7f45648d7640 1 monclient: found mon.noname-b 2024-03-29T11:14:08.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.558+0000 7f45648d7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:08.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.558+0000 7f45648d7640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:14:08.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.558+0000 7f45648d7640 1 -- 172.21.15.18:0/2650886721 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4560146340 con 0x7f456013f650 2024-03-29T11:14:08.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.558+0000 7f45648d7640 10 monclient: _finish_auth 0 2024-03-29T11:14:08.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.558+0000 7f45648d7640 10 monclient: _check_auth_tickets 2024-03-29T11:14:08.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.558+0000 7f45648d7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:08.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.558+0000 7f45648d7640 1 -- 172.21.15.18:0/2650886721 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f4550001690 con 0x7f456013f650 2024-03-29T11:14:08.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.558+0000 7f4567b3d640 5 monclient: authenticate success, global_id 4442 2024-03-29T11:14:08.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.558+0000 7f4567b3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:08.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.558+0000 7f45648d7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:08.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.558+0000 7f45648d7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:08.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.558+0000 7f45648d7640 10 monclient: dump: 2024-03-29T11:14:08.545 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:14:08.545 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:08.545 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:08.546 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:08.546 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:08.546 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:08.546 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:14:08.546 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:14:08.546 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:14:08.546 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:08.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.558+0000 7f45648d7640 1 -- 172.21.15.18:0/2650886721 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45500018b0 con 0x7f456013f650 2024-03-29T11:14:08.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.558+0000 7f45648d7640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:08.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.558+0000 7f45648d7640 1 -- 172.21.15.18:0/2650886721 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f4550001c40 con 0x7f456013f650 2024-03-29T11:14:08.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.558+0000 7f45648d7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:08.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.558+0000 7f45648d7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:08.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.558+0000 7f45648d7640 10 monclient: dump: 2024-03-29T11:14:08.546 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:14:08.547 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:08.547 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:08.547 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:08.547 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:08.547 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:08.547 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:14:08.547 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:14:08.547 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:14:08.547 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:08.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.558+0000 7f4567b3d640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:08.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.558+0000 7f4567b3d640 4 set_mon_vals no callback set 2024-03-29T11:14:08.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.559+0000 7f4567b3d640 10 monclient: shutdown 2024-03-29T11:14:08.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.559+0000 7f4567b3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:08.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.559+0000 7f4567b3d640 1 -- 172.21.15.18:0/2650886721 >> v1:172.21.15.18:6790/0 conn(0x7f456013f650 legacy=0x7f45600b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:08.547 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.559+0000 7f4567b3d640 1 -- 172.21.15.18:0/2650886721 shutdown_connections 2024-03-29T11:14:08.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.559+0000 7f4567b3d640 1 -- 172.21.15.18:0/2650886721 >> 172.21.15.18:0/2650886721 conn(0x7f4560066a00 msgr2=0x7f4560066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:08.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.561+0000 7f4567b3d640 1 -- 172.21.15.18:0/2650886721 shutdown_connections 2024-03-29T11:14:08.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.561+0000 7f4567b3d640 1 -- 172.21.15.18:0/2650886721 wait complete. 2024-03-29T11:14:08.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.562+0000 7f4567b3d640 10 monclient: build_initial_monmap 2024-03-29T11:14:08.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.562+0000 7f4567b3d640 10 monclient: monmap: 2024-03-29T11:14:08.548 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:14:08.548 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:08.548 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:14:08.548 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:14:08.548 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:08.548 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:08.548 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:14:08.548 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:14:08.548 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:14:08.549 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:08.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.563+0000 7f4567b3d640 1 librados: starting msgr at 2024-03-29T11:14:08.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.563+0000 7f4567b3d640 1 librados: starting objecter 2024-03-29T11:14:08.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.563+0000 7f4567b3d640 1 Processor -- start 2024-03-29T11:14:08.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.563+0000 7f4567b3d640 1 -- start start 2024-03-29T11:14:08.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.563+0000 7f4567b3d640 1 librados: setting wanted keys 2024-03-29T11:14:08.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.563+0000 7f4567b3d640 1 librados: calling monclient init 2024-03-29T11:14:08.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.563+0000 7f4567b3d640 10 monclient: init 2024-03-29T11:14:08.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.564+0000 7f4567b3d640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:08.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.564+0000 7f4567b3d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:14:08.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.564+0000 7f4567b3d640 10 monclient(hunting): picked mon.noname-a con 0x7f45600a9d70 addr v1:172.21.15.18:6789/0 2024-03-29T11:14:08.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.564+0000 7f4567b3d640 10 monclient(hunting): picked mon.noname-c con 0x7f456013f650 addr v1:172.21.15.18:6790/0 2024-03-29T11:14:08.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.564+0000 7f4567b3d640 10 monclient(hunting): picked mon.noname-b con 0x7f45601457f0 addr v1:172.21.15.70:6789/0 2024-03-29T11:14:08.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.564+0000 7f4567b3d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4560137c50 con 0x7f45600a9d70 2024-03-29T11:14:08.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.564+0000 7f4567b3d640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4560142d90 con 0x7f45601457f0 2024-03-29T11:14:08.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.564+0000 7f4567b3d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45600ac160 con 0x7f456013f650 2024-03-29T11:14:08.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.564+0000 7f4567b3d640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:08.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.564+0000 7f4567b3d640 10 monclient(hunting): authenticate will time out at 1074.836818s 2024-03-29T11:14:08.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.571+0000 7f45650d8640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f456013f650 0x7f4560136e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:59485/0 (socket says 172.21.15.18:59485) 2024-03-29T11:14:08.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.571+0000 7f45650d8640 1 -- 172.21.15.18:0/1410733953 learned_addr learned my addr 172.21.15.18:0/1410733953 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:14:08.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.571+0000 7f45567fc640 10 client.?.objecter ms_handle_connect 0x7f45600a9d70 2024-03-29T11:14:08.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.571+0000 7f45567fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:08.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.571+0000 7f45567fc640 10 client.?.objecter ms_handle_connect 0x7f456013f650 2024-03-29T11:14:08.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.571+0000 7f45567fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:08.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.571+0000 7f45567fc640 10 client.?.objecter ms_handle_connect 0x7f45601457f0 2024-03-29T11:14:08.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.571+0000 7f45567fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:08.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.571+0000 7f45567fc640 1 -- 172.21.15.18:0/1410733953 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2930354136 0 0) 0x7f4560137c50 con 0x7f45600a9d70 2024-03-29T11:14:08.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.571+0000 7f45567fc640 10 monclient(hunting): _negotiate 2024-03-29T11:14:08.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.571+0000 7f45567fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:08.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.571+0000 7f45567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:08.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.571+0000 7f45567fc640 10 monclient(hunting): my global_id is 4389 2024-03-29T11:14:08.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.571+0000 7f45567fc640 1 -- 172.21.15.18:0/1410733953 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4544004eb0 con 0x7f45600a9d70 2024-03-29T11:14:08.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.571+0000 7f45567fc640 1 -- 172.21.15.18:0/1410733953 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 443027180 0 0) 0x7f4560142d90 con 0x7f45601457f0 2024-03-29T11:14:08.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.572+0000 7f45567fc640 10 monclient(hunting): _negotiate 2024-03-29T11:14:08.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.572+0000 7f45567fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:08.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.572+0000 7f45567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:08.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.572+0000 7f45567fc640 10 monclient(hunting): my global_id is 4378 2024-03-29T11:14:08.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.572+0000 7f45567fc640 1 -- 172.21.15.18:0/1410733953 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4560137c50 con 0x7f45601457f0 2024-03-29T11:14:08.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.572+0000 7f45567fc640 1 -- 172.21.15.18:0/1410733953 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1000380581 0 0) 0x7f45600ac160 con 0x7f456013f650 2024-03-29T11:14:08.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.572+0000 7f45567fc640 10 monclient(hunting): _negotiate 2024-03-29T11:14:08.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.572+0000 7f45567fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:08.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.572+0000 7f45567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:08.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.572+0000 7f45567fc640 10 monclient(hunting): my global_id is 4445 2024-03-29T11:14:08.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.572+0000 7f45567fc640 1 -- 172.21.15.18:0/1410733953 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4560142d90 con 0x7f456013f650 2024-03-29T11:14:08.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.572+0000 7f45567fc640 1 -- 172.21.15.18:0/1410733953 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1466138729 0 0) 0x7f4544004eb0 con 0x7f45600a9d70 2024-03-29T11:14:08.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.572+0000 7f45567fc640 1 -- 172.21.15.18:0/1410733953 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45600ac160 con 0x7f45600a9d70 2024-03-29T11:14:08.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.572+0000 7f45567fc640 1 -- 172.21.15.18:0/1410733953 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f455c001910 con 0x7f45600a9d70 2024-03-29T11:14:08.563 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.572+0000 7f45567fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:08.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.572+0000 7f45567fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:14:08.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.572+0000 7f45567fc640 10 monclient(hunting): dump: 2024-03-29T11:14:08.565 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:14:08.565 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:08.565 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:08.565 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:08.565 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:08.565 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:08.566 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:14:08.566 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:14:08.566 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:14:08.566 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:08.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.572+0000 7f45567fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:08.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.572+0000 7f45567fc640 1 -- 172.21.15.18:0/1410733953 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3110533268 0 0) 0x7f4560137c50 con 0x7f45601457f0 2024-03-29T11:14:08.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.572+0000 7f45567fc640 1 -- 172.21.15.18:0/1410733953 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4544004eb0 con 0x7f45601457f0 2024-03-29T11:14:08.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.572+0000 7f45567fc640 1 -- 172.21.15.18:0/1410733953 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f4558001810 con 0x7f45601457f0 2024-03-29T11:14:08.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.572+0000 7f45567fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:08.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.572+0000 7f45567fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:08.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.572+0000 7f45567fc640 10 monclient(hunting): dump: 2024-03-29T11:14:08.566 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:14:08.566 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:08.566 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:08.567 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:08.567 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:08.567 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:08.567 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:14:08.567 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:14:08.567 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:14:08.567 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:08.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.572+0000 7f45567fc640 1 -- 172.21.15.18:0/1410733953 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2921496419 0 0) 0x7f4560142d90 con 0x7f456013f650 2024-03-29T11:14:08.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.572+0000 7f45567fc640 1 -- 172.21.15.18:0/1410733953 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4560137c50 con 0x7f456013f650 2024-03-29T11:14:08.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.572+0000 7f45567fc640 1 -- 172.21.15.18:0/1410733953 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2754449666 0 0) 0x7f45600ac160 con 0x7f45600a9d70 2024-03-29T11:14:08.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.572+0000 7f45567fc640 1 -- 172.21.15.18:0/1410733953 >> v1:172.21.15.18:6790/0 conn(0x7f456013f650 legacy=0x7f4560136e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:08.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.573+0000 7f45567fc640 1 -- 172.21.15.18:0/1410733953 >> v1:172.21.15.70:6789/0 conn(0x7f45601457f0 legacy=0x7f4560137540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:08.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.573+0000 7f45567fc640 10 monclient: _finish_hunting 0 2024-03-29T11:14:08.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.573+0000 7f45567fc640 1 monclient: found mon.a 2024-03-29T11:14:08.567 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.573+0000 7f45567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:08.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.573+0000 7f45567fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:14:08.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.573+0000 7f45567fc640 1 -- 172.21.15.18:0/1410733953 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45600ac440 con 0x7f45600a9d70 2024-03-29T11:14:08.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.573+0000 7f45567fc640 10 monclient: _finish_auth 0 2024-03-29T11:14:08.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.573+0000 7f45567fc640 10 monclient: _check_auth_tickets 2024-03-29T11:14:08.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.573+0000 7f45567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:08.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.573+0000 7f4567b3d640 5 monclient: authenticate success, global_id 4389 2024-03-29T11:14:08.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.573+0000 7f4567b3d640 10 monclient: _renew_subs 2024-03-29T11:14:08.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.573+0000 7f4567b3d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:14:08.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.573+0000 7f4567b3d640 1 -- 172.21.15.18:0/1410733953 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4560141d10 con 0x7f45600a9d70 2024-03-29T11:14:08.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.573+0000 7f4567b3d640 10 client.4389.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:08.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.573+0000 7f4567b3d640 10 monclient: _renew_subs 2024-03-29T11:14:08.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.573+0000 7f4567b3d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:14:08.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.573+0000 7f4567b3d640 1 -- 172.21.15.18:0/1410733953 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f456013fda0 con 0x7f45600a9d70 2024-03-29T11:14:08.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.573+0000 7f4567b3d640 1 librados: init done 2024-03-29T11:14:08.568 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.573+0000 7f45567fc640 1 -- 172.21.15.18:0/1410733953 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f455c0047e0 con 0x7f45600a9d70 2024-03-29T11:14:08.569 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.573+0000 7f45567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:08.569 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.573+0000 7f45567fc640 1 -- 172.21.15.18:0/1410733953 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f455c004af0 con 0x7f45600a9d70 2024-03-29T11:14:08.569 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.573+0000 7f45567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:08.569 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.573+0000 7f45567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:14:08.569 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.573+0000 7f4557fff640 4 set_mon_vals no callback set 2024-03-29T11:14:08.569 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.573+0000 7f45567fc640 10 monclient: dump: 2024-03-29T11:14:08.569 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:14:08.569 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:08.569 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:08.569 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:08.569 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:08.569 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:08.569 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:14:08.569 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:14:08.569 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:14:08.570 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:08.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.577+0000 7f4567b3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:08.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.577+0000 7f4567b3d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:08.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.577+0000 7f4567b3d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:14:08.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.577+0000 7f4567b3d640 1 -- 172.21.15.18:0/1410733953 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4548005080 con 0x7f45600a9d70 2024-03-29T11:14:08.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.579+0000 7f45567fc640 1 -- 172.21.15.18:0/1410733953 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f455c01aac0 con 0x7f45600a9d70 2024-03-29T11:14:08.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.580+0000 7f45567fc640 1 -- 172.21.15.18:0/1410733953 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f455c0849f0 con 0x7f45600a9d70 2024-03-29T11:14:08.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.580+0000 7f45567fc640 10 client.4389.objecter ms_dispatch 0x7f4560000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:08.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.580+0000 7f45567fc640 3 client.4389.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:08.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.580+0000 7f45567fc640 3 client.4389.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:08.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.581+0000 7f45567fc640 20 client.4389.objecter dump_active .. 0 homeless 2024-03-29T11:14:08.570 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.581+0000 7f45650d8640 10 monclient: get_auth_request con 0x7f454406e050 auth_method 0 2024-03-29T11:14:08.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.584+0000 7f45567fc640 10 client.4389.objecter ms_handle_connect 0x7f454406e050 2024-03-29T11:14:08.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.588+0000 7f45567fc640 1 -- 172.21.15.18:0/1410733953 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f455c0502d0 con 0x7f45600a9d70 2024-03-29T11:14:08.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.588+0000 7f45567fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:08.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.588+0000 7f45567fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:08.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.908+0000 7f4567b3d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:08.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.908+0000 7f4567b3d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:08.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.908+0000 7f4567b3d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:14:08.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.908+0000 7f4567b3d640 1 -- 172.21.15.18:0/1410733953 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4548005330 con 0x7f45600a9d70 2024-03-29T11:14:08.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.909+0000 7f45567fc640 1 -- 172.21.15.18:0/1410733953 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f455c052200 con 0x7f45600a9d70 2024-03-29T11:14:08.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.909+0000 7f45567fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:08.887 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.909+0000 7f45567fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:08.888 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:14:08.888 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:08.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.914+0000 7f4567b3d640 10 librados: watch_flush enter 2024-03-29T11:14:08.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.914+0000 7f4567b3d640 10 librados: watch_flush exit 2024-03-29T11:14:08.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.914+0000 7f4567b3d640 20 client.4389.objecter shutdown clearing up homeless session... 2024-03-29T11:14:08.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.914+0000 7f4567b3d640 10 client.4389.objecter successfully canceled tick 2024-03-29T11:14:08.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.914+0000 7f4567b3d640 1 -- 172.21.15.18:0/1410733953 >> v1:172.21.15.70:6816/3101871562 conn(0x7f454406e050 legacy=0x7f4544070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:08.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.914+0000 7f4567b3d640 10 monclient: shutdown 2024-03-29T11:14:08.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.914+0000 7f4567b3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:08.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.914+0000 7f4567b3d640 1 -- 172.21.15.18:0/1410733953 >> v1:172.21.15.18:6789/0 conn(0x7f45600a9d70 legacy=0x7f4560142680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:08.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.915+0000 7f4567b3d640 1 -- 172.21.15.18:0/1410733953 shutdown_connections 2024-03-29T11:14:08.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.915+0000 7f4567b3d640 1 -- 172.21.15.18:0/1410733953 >> 172.21.15.18:0/1410733953 conn(0x7f4560066a00 msgr2=0x7f4560066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:08.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.915+0000 7f4567b3d640 1 -- 172.21.15.18:0/1410733953 shutdown_connections 2024-03-29T11:14:08.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.915+0000 7f4567b3d640 1 -- 172.21.15.18:0/1410733953 wait complete. 2024-03-29T11:14:08.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:08.916+0000 7f4567b3d640 1 librados: shutdown 2024-03-29T11:14:08.920 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:15.728 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:15.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.899+0000 7fd29ae4a640 10 monclient: get_monmap_and_config 2024-03-29T11:14:15.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.899+0000 7fd29ae4a640 10 monclient: build_initial_monmap 2024-03-29T11:14:15.895 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.899+0000 7fd29ae4a640 10 monclient: monmap: 2024-03-29T11:14:15.895 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:14:15.895 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:15.895 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:14:15.900354+0000 2024-03-29T11:14:15.895 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:14:15.900354+0000 2024-03-29T11:14:15.895 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:15.895 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:15.895 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:14:15.895 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:14:15.895 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:14:15.896 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:15.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.900+0000 7fd29ae4a640 1 Processor -- start 2024-03-29T11:14:15.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.900+0000 7fd29ae4a640 1 -- start start 2024-03-29T11:14:15.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.900+0000 7fd29ae4a640 10 monclient: init 2024-03-29T11:14:15.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.901+0000 7fd29ae4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:15.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.901+0000 7fd29ae4a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:14:15.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.901+0000 7fd29ae4a640 10 monclient(hunting): picked mon.noname-a con 0x7fd2940ad6c0 addr v1:172.21.15.18:6789/0 2024-03-29T11:14:15.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.901+0000 7fd29ae4a640 10 monclient(hunting): picked mon.noname-c con 0x7fd2940af6c0 addr v1:172.21.15.70:6789/0 2024-03-29T11:14:15.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.901+0000 7fd29ae4a640 10 monclient(hunting): picked mon.noname-b con 0x7fd29414fdc0 addr v1:172.21.15.18:6790/0 2024-03-29T11:14:15.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.901+0000 7fd29ae4a640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2940a94e0 con 0x7fd2940ad6c0 2024-03-29T11:14:15.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.901+0000 7fd29ae4a640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2940af3a0 con 0x7fd2940af6c0 2024-03-29T11:14:15.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.901+0000 7fd29ae4a640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2940afb60 con 0x7fd29414fdc0 2024-03-29T11:14:15.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.901+0000 7fd29ae4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:15.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.901+0000 7fd29ae4a640 10 monclient(hunting): authenticate will time out at 1082.173883s 2024-03-29T11:14:15.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.904+0000 7fd293fff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fd2940af6c0 0x7fd294152980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.18:49307/0 (socket says 172.21.15.18:49307) 2024-03-29T11:14:15.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.904+0000 7fd293fff640 1 -- 172.21.15.18:0/3486914276 learned_addr learned my addr 172.21.15.18:0/3486914276 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:14:15.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.904+0000 7fd2937fe640 1 -- 172.21.15.18:0/3486914276 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3171875140 0 0) 0x7fd2940af3a0 con 0x7fd2940af6c0 2024-03-29T11:14:15.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.904+0000 7fd2937fe640 10 monclient(hunting): _negotiate 2024-03-29T11:14:15.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.904+0000 7fd2937fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:15.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.904+0000 7fd2937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:15.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.904+0000 7fd2937fe640 10 monclient(hunting): my global_id is 4381 2024-03-29T11:14:15.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.904+0000 7fd2937fe640 1 -- 172.21.15.18:0/3486914276 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd284003520 con 0x7fd2940af6c0 2024-03-29T11:14:15.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.905+0000 7fd2937fe640 1 -- 172.21.15.18:0/3486914276 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3623063309 0 0) 0x7fd284003520 con 0x7fd2940af6c0 2024-03-29T11:14:15.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.915+0000 7fd2937fe640 1 -- 172.21.15.18:0/3486914276 >> v1:172.21.15.18:6790/0 conn(0x7fd29414fdc0 legacy=0x7fd29415a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:14:15.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.915+0000 7fd2937fe640 1 -- 172.21.15.18:0/3486914276 >> v1:172.21.15.18:6789/0 conn(0x7fd2940ad6c0 legacy=0x7fd29414f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:14:15.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.915+0000 7fd2937fe640 10 monclient: _finish_hunting 0 2024-03-29T11:14:15.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.915+0000 7fd2937fe640 1 monclient: found mon.noname-c 2024-03-29T11:14:15.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.915+0000 7fd2937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:15.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.915+0000 7fd2937fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.70:6789/0 2024-03-29T11:14:15.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.915+0000 7fd2937fe640 1 -- 172.21.15.18:0/3486914276 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2940aab80 con 0x7fd2940af6c0 2024-03-29T11:14:15.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.915+0000 7fd2937fe640 10 monclient: _finish_auth 0 2024-03-29T11:14:15.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.915+0000 7fd2937fe640 10 monclient: _check_auth_tickets 2024-03-29T11:14:15.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.915+0000 7fd2937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:15.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.915+0000 7fd2937fe640 1 -- 172.21.15.18:0/3486914276 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fd280001690 con 0x7fd2940af6c0 2024-03-29T11:14:15.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.915+0000 7fd2937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:15.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.915+0000 7fd2937fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:14:15.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.915+0000 7fd2937fe640 10 monclient: dump: 2024-03-29T11:14:15.898 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:14:15.898 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:15.898 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:15.898 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:15.898 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:15.898 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:15.898 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:14:15.899 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:14:15.899 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:14:15.899 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:15.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.915+0000 7fd29ae4a640 5 monclient: authenticate success, global_id 4381 2024-03-29T11:14:15.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.915+0000 7fd29ae4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:15.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.916+0000 7fd2937fe640 1 -- 172.21.15.18:0/3486914276 <== mon.1 v1:172.21.15.70:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2800018f0 con 0x7fd2940af6c0 2024-03-29T11:14:15.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.916+0000 7fd2937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:15.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.916+0000 7fd2937fe640 1 -- 172.21.15.18:0/3486914276 <== mon.1 v1:172.21.15.70:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fd280001d40 con 0x7fd2940af6c0 2024-03-29T11:14:15.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.916+0000 7fd2937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:15.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.916+0000 7fd2937fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:15.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.916+0000 7fd2937fe640 10 monclient: dump: 2024-03-29T11:14:15.899 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:14:15.899 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:15.899 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:15.899 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:15.900 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:15.900 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:15.900 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:14:15.900 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:14:15.900 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:14:15.900 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:15.900 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.916+0000 7fd29ae4a640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:15.900 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.916+0000 7fd29ae4a640 4 set_mon_vals no callback set 2024-03-29T11:14:15.900 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.917+0000 7fd29ae4a640 10 monclient: shutdown 2024-03-29T11:14:15.900 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.917+0000 7fd29ae4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:15.900 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.917+0000 7fd29ae4a640 1 -- 172.21.15.18:0/3486914276 >> v1:172.21.15.70:6789/0 conn(0x7fd2940af6c0 legacy=0x7fd294152980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:15.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.917+0000 7fd29ae4a640 1 -- 172.21.15.18:0/3486914276 shutdown_connections 2024-03-29T11:14:15.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.917+0000 7fd29ae4a640 1 -- 172.21.15.18:0/3486914276 >> 172.21.15.18:0/3486914276 conn(0x7fd2940b35e0 msgr2=0x7fd2940b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:15.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.920+0000 7fd29ae4a640 1 -- 172.21.15.18:0/3486914276 shutdown_connections 2024-03-29T11:14:15.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.921+0000 7fd29ae4a640 1 -- 172.21.15.18:0/3486914276 wait complete. 2024-03-29T11:14:15.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.922+0000 7fd29ae4a640 10 monclient: build_initial_monmap 2024-03-29T11:14:15.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.922+0000 7fd29ae4a640 10 monclient: monmap: 2024-03-29T11:14:15.905 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:14:15.905 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:15.905 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:14:15.906 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:14:15.906 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:15.906 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:15.906 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:14:15.906 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:14:15.906 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:14:15.906 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:15.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.922+0000 7fd29ae4a640 1 librados: starting msgr at 2024-03-29T11:14:15.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.922+0000 7fd29ae4a640 1 librados: starting objecter 2024-03-29T11:14:15.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.923+0000 7fd29ae4a640 1 Processor -- start 2024-03-29T11:14:15.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.923+0000 7fd29ae4a640 1 -- start start 2024-03-29T11:14:15.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.923+0000 7fd29ae4a640 1 librados: setting wanted keys 2024-03-29T11:14:15.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.923+0000 7fd29ae4a640 1 librados: calling monclient init 2024-03-29T11:14:15.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.923+0000 7fd29ae4a640 10 monclient: init 2024-03-29T11:14:15.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.923+0000 7fd29ae4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:15.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.923+0000 7fd29ae4a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:14:15.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.924+0000 7fd29ae4a640 10 monclient(hunting): picked mon.noname-c con 0x7fd2940ad6c0 addr v1:172.21.15.18:6790/0 2024-03-29T11:14:15.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.924+0000 7fd29ae4a640 10 monclient(hunting): picked mon.noname-b con 0x7fd2940af6c0 addr v1:172.21.15.70:6789/0 2024-03-29T11:14:15.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.924+0000 7fd298be6640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fd2940ad6c0 0x7fd29406eaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:56906/0 (socket says 172.21.15.18:56906) 2024-03-29T11:14:15.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.924+0000 7fd298be6640 1 -- 172.21.15.18:0/1442641884 learned_addr learned my addr 172.21.15.18:0/1442641884 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:14:15.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.924+0000 7fd2917fa640 10 client.?.objecter ms_handle_connect 0x7fd2940ad6c0 2024-03-29T11:14:15.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.924+0000 7fd2917fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:15.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.925+0000 7fd29ae4a640 10 monclient(hunting): picked mon.noname-a con 0x7fd29414fdc0 addr v1:172.21.15.18:6789/0 2024-03-29T11:14:15.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.925+0000 7fd29ae4a640 1 -- 172.21.15.18:0/1442641884 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd29408a380 con 0x7fd29414fdc0 2024-03-29T11:14:15.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.925+0000 7fd29ae4a640 1 -- 172.21.15.18:0/1442641884 --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd294151ca0 con 0x7fd2940af6c0 2024-03-29T11:14:15.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.925+0000 7fd29ae4a640 1 -- 172.21.15.18:0/1442641884 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd29406fb90 con 0x7fd2940ad6c0 2024-03-29T11:14:15.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.925+0000 7fd29ae4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:15.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.925+0000 7fd29ae4a640 10 monclient(hunting): authenticate will time out at 1082.197011s 2024-03-29T11:14:15.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.925+0000 7fd2917fa640 1 -- 172.21.15.18:0/1442641884 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3492829345 0 0) 0x7fd29406fb90 con 0x7fd2940ad6c0 2024-03-29T11:14:15.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.925+0000 7fd2917fa640 10 monclient(hunting): _negotiate 2024-03-29T11:14:15.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.925+0000 7fd2917fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:15.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.925+0000 7fd2917fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:15.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.925+0000 7fd2917fa640 10 monclient(hunting): my global_id is 4448 2024-03-29T11:14:15.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.925+0000 7fd2917fa640 1 -- 172.21.15.18:0/1442641884 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd278004eb0 con 0x7fd2940ad6c0 2024-03-29T11:14:15.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.926+0000 7fd2917fa640 1 -- 172.21.15.18:0/1442641884 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2053401896 0 0) 0x7fd278004eb0 con 0x7fd2940ad6c0 2024-03-29T11:14:15.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.926+0000 7fd2917fa640 1 -- 172.21.15.18:0/1442641884 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd29406fb90 con 0x7fd2940ad6c0 2024-03-29T11:14:15.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.926+0000 7fd2917fa640 1 -- 172.21.15.18:0/1442641884 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fd288001960 con 0x7fd2940ad6c0 2024-03-29T11:14:15.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.926+0000 7fd2917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:15.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.926+0000 7fd2917fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:14:15.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.926+0000 7fd2917fa640 10 monclient(hunting): dump: 2024-03-29T11:14:15.909 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:14:15.909 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:15.909 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:15.909 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:15.909 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:15.909 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:15.909 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:14:15.909 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:14:15.909 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:14:15.909 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:15.909 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.926+0000 7fd2917fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:15.909 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.926+0000 7fd2917fa640 1 -- 172.21.15.18:0/1442641884 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1396758349 0 0) 0x7fd29406fb90 con 0x7fd2940ad6c0 2024-03-29T11:14:15.909 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.926+0000 7fd2917fa640 1 -- 172.21.15.18:0/1442641884 >> v1:172.21.15.70:6789/0 conn(0x7fd2940af6c0 legacy=0x7fd294070570 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:14:15.909 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.926+0000 7fd2917fa640 1 -- 172.21.15.18:0/1442641884 >> v1:172.21.15.18:6789/0 conn(0x7fd29414fdc0 legacy=0x7fd2941f77e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:14:15.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.929+0000 7fd2917fa640 10 monclient: _finish_hunting 0 2024-03-29T11:14:15.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.929+0000 7fd2917fa640 1 monclient: found mon.c 2024-03-29T11:14:15.912 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.929+0000 7fd2917fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:15.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.929+0000 7fd2917fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:14:15.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.929+0000 7fd2917fa640 1 -- 172.21.15.18:0/1442641884 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd29406fe70 con 0x7fd2940ad6c0 2024-03-29T11:14:15.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.929+0000 7fd2917fa640 10 monclient: _finish_auth 0 2024-03-29T11:14:15.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.929+0000 7fd2917fa640 10 monclient: _check_auth_tickets 2024-03-29T11:14:15.913 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.929+0000 7fd2917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:15.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.929+0000 7fd29ae4a640 5 monclient: authenticate success, global_id 4448 2024-03-29T11:14:15.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.929+0000 7fd29ae4a640 10 monclient: _renew_subs 2024-03-29T11:14:15.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.929+0000 7fd29ae4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:14:15.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.929+0000 7fd29ae4a640 1 -- 172.21.15.18:0/1442641884 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd294070150 con 0x7fd2940ad6c0 2024-03-29T11:14:15.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.929+0000 7fd29ae4a640 10 client.4448.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:15.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.929+0000 7fd29ae4a640 10 monclient: _renew_subs 2024-03-29T11:14:15.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.929+0000 7fd29ae4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:14:15.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.929+0000 7fd29ae4a640 1 -- 172.21.15.18:0/1442641884 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd294144810 con 0x7fd2940ad6c0 2024-03-29T11:14:15.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.929+0000 7fd29ae4a640 1 librados: init done 2024-03-29T11:14:15.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.930+0000 7fd29ae4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:15.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.930+0000 7fd29ae4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:15.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.931+0000 7fd29ae4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:14:15.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.931+0000 7fd29ae4a640 1 -- 172.21.15.18:0/1442641884 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd294144b60 con 0x7fd2940ad6c0 2024-03-29T11:14:15.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.931+0000 7fd2917fa640 1 -- 172.21.15.18:0/1442641884 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd288001c80 con 0x7fd2940ad6c0 2024-03-29T11:14:15.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.931+0000 7fd2917fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:15.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.931+0000 7fd2917fa640 1 -- 172.21.15.18:0/1442641884 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fd288003a40 con 0x7fd2940ad6c0 2024-03-29T11:14:15.916 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.931+0000 7fd2917fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:15.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.931+0000 7fd2917fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:15.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.931+0000 7fd2917fa640 10 monclient: dump: 2024-03-29T11:14:15.917 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:14:15.917 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:15.917 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:15.917 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:15.917 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:15.917 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:15.917 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:14:15.917 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:14:15.917 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:14:15.918 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:15.918 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.931+0000 7fd292ffd640 4 set_mon_vals no callback set 2024-03-29T11:14:15.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.935+0000 7fd2917fa640 1 -- 172.21.15.18:0/1442641884 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7fd288003ba0 con 0x7fd2940ad6c0 2024-03-29T11:14:15.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.936+0000 7fd2917fa640 1 -- 172.21.15.18:0/1442641884 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7fd288084a90 con 0x7fd2940ad6c0 2024-03-29T11:14:15.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.936+0000 7fd2917fa640 10 client.4448.objecter ms_dispatch 0x7fd294000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:15.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.936+0000 7fd2917fa640 3 client.4448.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:15.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.936+0000 7fd2917fa640 3 client.4448.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:15.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.936+0000 7fd2917fa640 20 client.4448.objecter dump_active .. 0 homeless 2024-03-29T11:14:15.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.936+0000 7fd293fff640 10 monclient: get_auth_request con 0x7fd27806dcd0 auth_method 0 2024-03-29T11:14:15.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.938+0000 7fd2917fa640 10 client.4448.objecter ms_handle_connect 0x7fd27806dcd0 2024-03-29T11:14:15.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.942+0000 7fd2917fa640 1 -- 172.21.15.18:0/1442641884 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd288050370 con 0x7fd2940ad6c0 2024-03-29T11:14:15.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.942+0000 7fd2917fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:15.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:15.942+0000 7fd2917fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:16.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:16.270+0000 7fd29ae4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:16.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:16.271+0000 7fd29ae4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:16.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:16.271+0000 7fd29ae4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:14:16.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:16.271+0000 7fd29ae4a640 1 -- 172.21.15.18:0/1442641884 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd294075f10 con 0x7fd2940ad6c0 2024-03-29T11:14:16.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:16.272+0000 7fd2917fa640 1 -- 172.21.15.18:0/1442641884 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7fd2880522a0 con 0x7fd2940ad6c0 2024-03-29T11:14:16.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:16.272+0000 7fd2917fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:16.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:16.272+0000 7fd2917fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:16.251 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:14:16.251 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:16.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:16.277+0000 7fd29ae4a640 10 librados: watch_flush enter 2024-03-29T11:14:16.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:16.277+0000 7fd29ae4a640 10 librados: watch_flush exit 2024-03-29T11:14:16.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:16.277+0000 7fd29ae4a640 20 client.4448.objecter shutdown clearing up homeless session... 2024-03-29T11:14:16.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:16.277+0000 7fd29ae4a640 10 client.4448.objecter successfully canceled tick 2024-03-29T11:14:16.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:16.277+0000 7fd29ae4a640 1 -- 172.21.15.18:0/1442641884 >> v1:172.21.15.70:6816/3101871562 conn(0x7fd27806dcd0 legacy=0x7fd278070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:16.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:16.277+0000 7fd29ae4a640 10 monclient: shutdown 2024-03-29T11:14:16.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:16.277+0000 7fd29ae4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:16.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:16.277+0000 7fd29ae4a640 1 -- 172.21.15.18:0/1442641884 >> v1:172.21.15.18:6790/0 conn(0x7fd2940ad6c0 legacy=0x7fd29406eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:16.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:16.277+0000 7fd29ae4a640 1 -- 172.21.15.18:0/1442641884 shutdown_connections 2024-03-29T11:14:16.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:16.277+0000 7fd29ae4a640 1 -- 172.21.15.18:0/1442641884 >> 172.21.15.18:0/1442641884 conn(0x7fd2940b35e0 msgr2=0x7fd2940a9e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:16.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:16.277+0000 7fd29ae4a640 1 -- 172.21.15.18:0/1442641884 shutdown_connections 2024-03-29T11:14:16.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:16.277+0000 7fd29ae4a640 1 -- 172.21.15.18:0/1442641884 wait complete. 2024-03-29T11:14:16.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:16.277+0000 7fd29ae4a640 1 librados: shutdown 2024-03-29T11:14:16.272 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:23.080 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:23.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.257+0000 7f0bd3f6b640 10 monclient: get_monmap_and_config 2024-03-29T11:14:23.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.257+0000 7f0bd3f6b640 10 monclient: build_initial_monmap 2024-03-29T11:14:23.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.257+0000 7f0bd3f6b640 10 monclient: monmap: 2024-03-29T11:14:23.235 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:14:23.235 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:23.235 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:14:23.258138+0000 2024-03-29T11:14:23.235 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:14:23.258138+0000 2024-03-29T11:14:23.235 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:23.235 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:23.235 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:14:23.235 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:14:23.235 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:14:23.236 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:23.237 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.259+0000 7f0bd3f6b640 1 Processor -- start 2024-03-29T11:14:23.237 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.259+0000 7f0bd3f6b640 1 -- start start 2024-03-29T11:14:23.237 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.259+0000 7f0bd3f6b640 10 monclient: init 2024-03-29T11:14:23.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.260+0000 7f0bd3f6b640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:23.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.260+0000 7f0bd3f6b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:14:23.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.260+0000 7f0bd3f6b640 10 monclient(hunting): picked mon.noname-a con 0x7f0bcc0b45a0 addr v1:172.21.15.18:6789/0 2024-03-29T11:14:23.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.260+0000 7f0bd3f6b640 10 monclient(hunting): picked mon.noname-b con 0x7f0bcc13fb10 addr v1:172.21.15.18:6790/0 2024-03-29T11:14:23.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.260+0000 7f0bd3f6b640 10 monclient(hunting): picked mon.noname-c con 0x7f0bcc0b3920 addr v1:172.21.15.70:6789/0 2024-03-29T11:14:23.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.261+0000 7f0bd3f6b640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bcc0b1de0 con 0x7f0bcc0b45a0 2024-03-29T11:14:23.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.261+0000 7f0bd3f6b640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bcc0b0670 con 0x7f0bcc0b3920 2024-03-29T11:14:23.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.261+0000 7f0bd3f6b640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bcc0af560 con 0x7f0bcc13fb10 2024-03-29T11:14:23.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.261+0000 7f0bd3f6b640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:23.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.261+0000 7f0bd3f6b640 10 monclient(hunting): authenticate will time out at 1089.533195s 2024-03-29T11:14:23.241 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.263+0000 7f0bd1506640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f0bcc13fb10 0x7f0bcc0b0fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:41694/0 (socket says 172.21.15.18:41694) 2024-03-29T11:14:23.242 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.263+0000 7f0bd1506640 1 -- 172.21.15.18:0/2595473608 learned_addr learned my addr 172.21.15.18:0/2595473608 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:14:23.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.265+0000 7f0bd0d05640 1 -- 172.21.15.18:0/2595473608 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3733036408 0 0) 0x7f0bcc0b0670 con 0x7f0bcc0b3920 2024-03-29T11:14:23.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.265+0000 7f0bd0d05640 10 monclient(hunting): _negotiate 2024-03-29T11:14:23.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.265+0000 7f0bd0d05640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:23.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.265+0000 7f0bd0d05640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:23.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.265+0000 7f0bd0d05640 10 monclient(hunting): my global_id is 4384 2024-03-29T11:14:23.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.265+0000 7f0bd0d05640 1 -- 172.21.15.18:0/2595473608 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bb8003520 con 0x7f0bcc0b3920 2024-03-29T11:14:23.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.266+0000 7f0bd0d05640 1 -- 172.21.15.18:0/2595473608 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1898309782 0 0) 0x7f0bcc0b1de0 con 0x7f0bcc0b45a0 2024-03-29T11:14:23.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.266+0000 7f0bd0d05640 10 monclient(hunting): _negotiate 2024-03-29T11:14:23.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.266+0000 7f0bd0d05640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:23.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.266+0000 7f0bd0d05640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:23.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.266+0000 7f0bd0d05640 10 monclient(hunting): my global_id is 4392 2024-03-29T11:14:23.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.266+0000 7f0bd0d05640 1 -- 172.21.15.18:0/2595473608 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bb8003840 con 0x7f0bcc0b45a0 2024-03-29T11:14:23.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.266+0000 7f0bd0d05640 1 -- 172.21.15.18:0/2595473608 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 222554176 0 0) 0x7f0bb8003520 con 0x7f0bcc0b3920 2024-03-29T11:14:23.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.266+0000 7f0bd0d05640 1 -- 172.21.15.18:0/2595473608 >> v1:172.21.15.18:6790/0 conn(0x7f0bcc13fb10 legacy=0x7f0bcc0b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:14:23.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.266+0000 7f0bd0d05640 1 -- 172.21.15.18:0/2595473608 >> v1:172.21.15.18:6789/0 conn(0x7f0bcc0b45a0 legacy=0x7f0bcc13f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:23.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.266+0000 7f0bd0d05640 10 monclient: _finish_hunting 0 2024-03-29T11:14:23.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.266+0000 7f0bd0d05640 1 monclient: found mon.noname-c 2024-03-29T11:14:23.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.266+0000 7f0bd0d05640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:23.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.266+0000 7f0bd0d05640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.70:6789/0 2024-03-29T11:14:23.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.266+0000 7f0bd0d05640 1 -- 172.21.15.18:0/2595473608 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bcc0b08c0 con 0x7f0bcc0b3920 2024-03-29T11:14:23.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.266+0000 7f0bd0d05640 10 monclient: _finish_auth 0 2024-03-29T11:14:23.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.266+0000 7f0bd0d05640 10 monclient: _check_auth_tickets 2024-03-29T11:14:23.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.266+0000 7f0bd0d05640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:23.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.266+0000 7f0bd0d05640 1 -- 172.21.15.18:0/2595473608 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f0bc40019f0 con 0x7f0bcc0b3920 2024-03-29T11:14:23.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.266+0000 7f0bd0d05640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:23.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.266+0000 7f0bd0d05640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:14:23.248 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.266+0000 7f0bd0d05640 10 monclient: dump: 2024-03-29T11:14:23.248 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:14:23.248 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:23.248 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:23.249 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:23.249 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:23.249 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:23.249 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:14:23.249 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:14:23.249 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:14:23.249 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:23.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.266+0000 7f0bd0d05640 1 -- 172.21.15.18:0/2595473608 <== mon.1 v1:172.21.15.70:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0bc4001bd0 con 0x7f0bcc0b3920 2024-03-29T11:14:23.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.266+0000 7f0bd0d05640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:23.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.267+0000 7f0bd0d05640 1 -- 172.21.15.18:0/2595473608 <== mon.1 v1:172.21.15.70:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f0bc40019f0 con 0x7f0bcc0b3920 2024-03-29T11:14:23.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.267+0000 7f0bd0d05640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:23.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.267+0000 7f0bd0d05640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:23.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.267+0000 7f0bd0d05640 10 monclient: dump: 2024-03-29T11:14:23.250 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:14:23.250 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:23.250 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:23.250 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:23.250 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:23.250 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:23.250 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:14:23.250 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:14:23.250 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:14:23.250 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:23.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.267+0000 7f0bd3f6b640 5 monclient: authenticate success, global_id 4384 2024-03-29T11:14:23.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.267+0000 7f0bd3f6b640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:23.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.267+0000 7f0bd3f6b640 4 set_mon_vals no callback set 2024-03-29T11:14:23.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.268+0000 7f0bd3f6b640 10 monclient: shutdown 2024-03-29T11:14:23.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.270+0000 7f0bd3f6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:23.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.270+0000 7f0bd3f6b640 1 -- 172.21.15.18:0/2595473608 >> v1:172.21.15.70:6789/0 conn(0x7f0bcc0b3920 legacy=0x7f0bcc0b16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:23.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.272+0000 7f0bd3f6b640 1 -- 172.21.15.18:0/2595473608 shutdown_connections 2024-03-29T11:14:23.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.273+0000 7f0bd3f6b640 1 -- 172.21.15.18:0/2595473608 >> 172.21.15.18:0/2595473608 conn(0x7f0bcc066a00 msgr2=0x7f0bcc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:23.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.273+0000 7f0bd3f6b640 1 -- 172.21.15.18:0/2595473608 shutdown_connections 2024-03-29T11:14:23.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.273+0000 7f0bd3f6b640 1 -- 172.21.15.18:0/2595473608 wait complete. 2024-03-29T11:14:23.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.274+0000 7f0bd3f6b640 10 monclient: build_initial_monmap 2024-03-29T11:14:23.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.274+0000 7f0bd3f6b640 10 monclient: monmap: 2024-03-29T11:14:23.255 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:14:23.255 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:23.255 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:14:23.255 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:14:23.255 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:23.255 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:23.255 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:14:23.256 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:14:23.256 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:14:23.256 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:23.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.275+0000 7f0bd3f6b640 1 librados: starting msgr at 2024-03-29T11:14:23.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.275+0000 7f0bd3f6b640 1 librados: starting objecter 2024-03-29T11:14:23.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.275+0000 7f0bd3f6b640 1 Processor -- start 2024-03-29T11:14:23.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.275+0000 7f0bd3f6b640 1 -- start start 2024-03-29T11:14:23.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.275+0000 7f0bd3f6b640 1 librados: setting wanted keys 2024-03-29T11:14:23.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.275+0000 7f0bd3f6b640 1 librados: calling monclient init 2024-03-29T11:14:23.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.275+0000 7f0bd3f6b640 10 monclient: init 2024-03-29T11:14:23.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.276+0000 7f0bd3f6b640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:23.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.276+0000 7f0bd3f6b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:14:23.259 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.279+0000 7f0bd3f6b640 10 monclient(hunting): picked mon.noname-a con 0x7f0bcc0b3920 addr v1:172.21.15.18:6789/0 2024-03-29T11:14:23.259 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.280+0000 7f0bd3f6b640 10 monclient(hunting): picked mon.noname-c con 0x7f0bcc0b45a0 addr v1:172.21.15.18:6790/0 2024-03-29T11:14:23.259 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.280+0000 7f0bd1d07640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f0bcc0b3920 0x7f0bcc1d6530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:45391/0 (socket says 172.21.15.18:45391) 2024-03-29T11:14:23.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.280+0000 7f0bd1d07640 1 -- 172.21.15.18:0/1046824284 learned_addr learned my addr 172.21.15.18:0/1046824284 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:14:23.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.280+0000 7f0bd3f6b640 10 monclient(hunting): picked mon.noname-b con 0x7f0bcc13fb10 addr v1:172.21.15.70:6789/0 2024-03-29T11:14:23.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.281+0000 7f0bd3f6b640 1 -- 172.21.15.18:0/1046824284 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bcc070590 con 0x7f0bcc0b3920 2024-03-29T11:14:23.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.281+0000 7f0bd3f6b640 1 -- 172.21.15.18:0/1046824284 --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bcc1d6010 con 0x7f0bcc13fb10 2024-03-29T11:14:23.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.281+0000 7f0bd3f6b640 1 -- 172.21.15.18:0/1046824284 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bcc070d10 con 0x7f0bcc0b45a0 2024-03-29T11:14:23.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.281+0000 7f0bd3f6b640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:23.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.281+0000 7f0bd3f6b640 10 monclient(hunting): authenticate will time out at 1089.553015s 2024-03-29T11:14:23.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.280+0000 7f0bc27fc640 10 client.?.objecter ms_handle_connect 0x7f0bcc0b3920 2024-03-29T11:14:23.260 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.281+0000 7f0bc27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:23.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.284+0000 7f0bc27fc640 10 client.?.objecter ms_handle_connect 0x7f0bcc0b45a0 2024-03-29T11:14:23.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.284+0000 7f0bc27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:23.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.284+0000 7f0bc27fc640 1 -- 172.21.15.18:0/1046824284 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 805829986 0 0) 0x7f0bcc070590 con 0x7f0bcc0b3920 2024-03-29T11:14:23.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.284+0000 7f0bc27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:14:23.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.284+0000 7f0bc27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:23.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.284+0000 7f0bc27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:23.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.284+0000 7f0bc27fc640 10 monclient(hunting): my global_id is 4395 2024-03-29T11:14:23.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.284+0000 7f0bc27fc640 1 -- 172.21.15.18:0/1046824284 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bb0004eb0 con 0x7f0bcc0b3920 2024-03-29T11:14:23.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.285+0000 7f0bc27fc640 10 client.?.objecter ms_handle_connect 0x7f0bcc13fb10 2024-03-29T11:14:23.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.285+0000 7f0bc27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:23.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.285+0000 7f0bc27fc640 1 -- 172.21.15.18:0/1046824284 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 415964571 0 0) 0x7f0bcc070d10 con 0x7f0bcc0b45a0 2024-03-29T11:14:23.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.285+0000 7f0bc27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:14:23.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.285+0000 7f0bc27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:23.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.285+0000 7f0bc27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:23.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.285+0000 7f0bc27fc640 10 monclient(hunting): my global_id is 4451 2024-03-29T11:14:23.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.285+0000 7f0bc27fc640 1 -- 172.21.15.18:0/1046824284 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bcc070590 con 0x7f0bcc0b45a0 2024-03-29T11:14:23.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.285+0000 7f0bc27fc640 1 -- 172.21.15.18:0/1046824284 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1532182764 0 0) 0x7f0bcc1d6010 con 0x7f0bcc13fb10 2024-03-29T11:14:23.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.285+0000 7f0bc27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:14:23.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.285+0000 7f0bc27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:23.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.285+0000 7f0bc27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:23.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.285+0000 7f0bc27fc640 10 monclient(hunting): my global_id is 4387 2024-03-29T11:14:23.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.285+0000 7f0bc27fc640 1 -- 172.21.15.18:0/1046824284 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0bcc070d10 con 0x7f0bcc13fb10 2024-03-29T11:14:23.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.285+0000 7f0bc27fc640 1 -- 172.21.15.18:0/1046824284 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1064408924 0 0) 0x7f0bb0004eb0 con 0x7f0bcc0b3920 2024-03-29T11:14:23.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.285+0000 7f0bc27fc640 1 -- 172.21.15.18:0/1046824284 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0bcc1d6010 con 0x7f0bcc0b3920 2024-03-29T11:14:23.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.285+0000 7f0bc27fc640 1 -- 172.21.15.18:0/1046824284 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f0bc8001a10 con 0x7f0bcc0b3920 2024-03-29T11:14:23.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.285+0000 7f0bc27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:23.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.285+0000 7f0bc27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:14:23.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.285+0000 7f0bc27fc640 10 monclient(hunting): dump: 2024-03-29T11:14:23.266 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:14:23.266 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:23.266 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:23.267 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:23.267 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:23.267 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:23.267 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:14:23.267 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:14:23.267 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:14:23.267 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:23.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.285+0000 7f0bc27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:23.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.285+0000 7f0bc27fc640 1 -- 172.21.15.18:0/1046824284 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3335406240 0 0) 0x7f0bcc070d10 con 0x7f0bcc13fb10 2024-03-29T11:14:23.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.286+0000 7f0bc27fc640 1 -- 172.21.15.18:0/1046824284 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0bb0004eb0 con 0x7f0bcc13fb10 2024-03-29T11:14:23.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.286+0000 7f0bc27fc640 1 -- 172.21.15.18:0/1046824284 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f0bc40019f0 con 0x7f0bcc13fb10 2024-03-29T11:14:23.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.286+0000 7f0bc27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:23.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.286+0000 7f0bc27fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:23.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.286+0000 7f0bc27fc640 10 monclient(hunting): dump: 2024-03-29T11:14:23.273 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:14:23.273 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:23.274 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:23.274 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:23.274 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:23.274 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:23.274 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:14:23.274 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:14:23.274 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:14:23.274 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:23.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.286+0000 7f0bc27fc640 1 -- 172.21.15.18:0/1046824284 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2662469969 0 0) 0x7f0bcc1d6010 con 0x7f0bcc0b3920 2024-03-29T11:14:23.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.286+0000 7f0bc27fc640 1 -- 172.21.15.18:0/1046824284 >> v1:172.21.15.18:6790/0 conn(0x7f0bcc0b45a0 legacy=0x7f0bcc136d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:23.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.286+0000 7f0bc27fc640 1 -- 172.21.15.18:0/1046824284 >> v1:172.21.15.70:6789/0 conn(0x7f0bcc13fb10 legacy=0x7f0bcc06e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:23.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.286+0000 7f0bc27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:14:23.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.286+0000 7f0bc27fc640 1 monclient: found mon.a 2024-03-29T11:14:23.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.286+0000 7f0bc27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:23.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.286+0000 7f0bc27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:14:23.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.286+0000 7f0bc27fc640 1 -- 172.21.15.18:0/1046824284 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bcc070ea0 con 0x7f0bcc0b3920 2024-03-29T11:14:23.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.286+0000 7f0bc27fc640 10 monclient: _finish_auth 0 2024-03-29T11:14:23.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.286+0000 7f0bc27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:14:23.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.286+0000 7f0bc27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:23.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.286+0000 7f0bd3f6b640 5 monclient: authenticate success, global_id 4395 2024-03-29T11:14:23.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.286+0000 7f0bd3f6b640 10 monclient: _renew_subs 2024-03-29T11:14:23.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.286+0000 7f0bd3f6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:14:23.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.286+0000 7f0bd3f6b640 1 -- 172.21.15.18:0/1046824284 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bcc1d6c40 con 0x7f0bcc0b3920 2024-03-29T11:14:23.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.286+0000 7f0bd3f6b640 10 client.4395.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:23.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.286+0000 7f0bd3f6b640 10 monclient: _renew_subs 2024-03-29T11:14:23.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.286+0000 7f0bd3f6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:14:23.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.286+0000 7f0bd3f6b640 1 -- 172.21.15.18:0/1046824284 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bcc1d6ef0 con 0x7f0bcc0b3920 2024-03-29T11:14:23.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.286+0000 7f0bd3f6b640 1 librados: init done 2024-03-29T11:14:23.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.287+0000 7f0bc27fc640 1 -- 172.21.15.18:0/1046824284 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0bc8001bf0 con 0x7f0bcc0b3920 2024-03-29T11:14:23.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.287+0000 7f0bc27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:23.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.287+0000 7f0bc3fff640 4 set_mon_vals no callback set 2024-03-29T11:14:23.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.289+0000 7f0bc27fc640 1 -- 172.21.15.18:0/1046824284 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f0bc8003a60 con 0x7f0bcc0b3920 2024-03-29T11:14:23.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.289+0000 7f0bc27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:23.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.289+0000 7f0bc27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:14:23.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.289+0000 7f0bc27fc640 10 monclient: dump: 2024-03-29T11:14:23.276 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:14:23.276 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:23.276 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:23.276 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:23.277 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:23.277 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:23.277 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:14:23.277 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:14:23.277 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:14:23.277 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:23.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.289+0000 7f0bc27fc640 1 -- 172.21.15.18:0/1046824284 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f0bc8003c80 con 0x7f0bcc0b3920 2024-03-29T11:14:23.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.290+0000 7f0bc27fc640 1 -- 172.21.15.18:0/1046824284 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f0bc8084ba0 con 0x7f0bcc0b3920 2024-03-29T11:14:23.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.290+0000 7f0bc27fc640 10 client.4395.objecter ms_dispatch 0x7f0bcc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:23.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.290+0000 7f0bc27fc640 3 client.4395.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:23.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.290+0000 7f0bc27fc640 3 client.4395.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:23.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.290+0000 7f0bc27fc640 20 client.4395.objecter dump_active .. 0 homeless 2024-03-29T11:14:23.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.290+0000 7f0bd1506640 10 monclient: get_auth_request con 0x7f0bb006dfd0 auth_method 0 2024-03-29T11:14:23.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.291+0000 7f0bd3f6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:23.278 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.291+0000 7f0bd3f6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:23.278 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.291+0000 7f0bc27fc640 10 client.4395.objecter ms_handle_connect 0x7f0bb006dfd0 2024-03-29T11:14:23.278 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.291+0000 7f0bd3f6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:14:23.278 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.291+0000 7f0bd3f6b640 1 -- 172.21.15.18:0/1046824284 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bb4005080 con 0x7f0bcc0b3920 2024-03-29T11:14:23.278 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.301+0000 7f0bc27fc640 1 -- 172.21.15.18:0/1046824284 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0bc8050480 con 0x7f0bcc0b3920 2024-03-29T11:14:23.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.301+0000 7f0bc27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:23.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.301+0000 7f0bc27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:23.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.657+0000 7f0bd3f6b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:23.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.657+0000 7f0bd3f6b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:23.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.657+0000 7f0bd3f6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:14:23.636 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.657+0000 7f0bd3f6b640 1 -- 172.21.15.18:0/1046824284 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0bb4005330 con 0x7f0bcc0b3920 2024-03-29T11:14:23.641 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.663+0000 7f0bc27fc640 1 -- 172.21.15.18:0/1046824284 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f0bc80523b0 con 0x7f0bcc0b3920 2024-03-29T11:14:23.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.663+0000 7f0bc27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:23.642 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.663+0000 7f0bc27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:23.642 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:14:23.642 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:23.646 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.667+0000 7f0bd3f6b640 10 librados: watch_flush enter 2024-03-29T11:14:23.646 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.667+0000 7f0bd3f6b640 10 librados: watch_flush exit 2024-03-29T11:14:23.646 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.667+0000 7f0bd3f6b640 20 client.4395.objecter shutdown clearing up homeless session... 2024-03-29T11:14:23.646 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.667+0000 7f0bd3f6b640 10 client.4395.objecter successfully canceled tick 2024-03-29T11:14:23.646 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.668+0000 7f0bd3f6b640 1 -- 172.21.15.18:0/1046824284 >> v1:172.21.15.70:6816/3101871562 conn(0x7f0bb006dfd0 legacy=0x7f0bb0070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:23.646 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.668+0000 7f0bd3f6b640 10 monclient: shutdown 2024-03-29T11:14:23.646 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.668+0000 7f0bd3f6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:23.646 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.668+0000 7f0bd3f6b640 1 -- 172.21.15.18:0/1046824284 >> v1:172.21.15.18:6789/0 conn(0x7f0bcc0b3920 legacy=0x7f0bcc1d6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:23.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.668+0000 7f0bd3f6b640 1 -- 172.21.15.18:0/1046824284 shutdown_connections 2024-03-29T11:14:23.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.668+0000 7f0bd3f6b640 1 -- 172.21.15.18:0/1046824284 >> 172.21.15.18:0/1046824284 conn(0x7f0bcc066a00 msgr2=0x7f0bcc138200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:23.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.669+0000 7f0bd3f6b640 1 -- 172.21.15.18:0/1046824284 shutdown_connections 2024-03-29T11:14:23.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.669+0000 7f0bd3f6b640 1 -- 172.21.15.18:0/1046824284 wait complete. 2024-03-29T11:14:23.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:23.669+0000 7f0bd3f6b640 1 librados: shutdown 2024-03-29T11:14:23.665 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:30.472 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:30.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.632+0000 7fc92594b640 10 monclient: get_monmap_and_config 2024-03-29T11:14:30.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.632+0000 7fc92594b640 10 monclient: build_initial_monmap 2024-03-29T11:14:30.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.632+0000 7fc92594b640 10 monclient: monmap: 2024-03-29T11:14:30.611 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:14:30.611 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:30.611 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:14:30.633307+0000 2024-03-29T11:14:30.611 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:14:30.633307+0000 2024-03-29T11:14:30.611 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:30.611 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:30.612 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:14:30.612 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:14:30.612 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:14:30.612 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:30.614 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.634+0000 7fc92594b640 1 Processor -- start 2024-03-29T11:14:30.614 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.634+0000 7fc92594b640 1 -- start start 2024-03-29T11:14:30.614 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.634+0000 7fc92594b640 10 monclient: init 2024-03-29T11:14:30.614 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.635+0000 7fc92594b640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:30.614 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.635+0000 7fc92594b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:14:30.614 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.635+0000 7fc92594b640 10 monclient(hunting): picked mon.noname-b con 0x7fc9200acf60 addr v1:172.21.15.18:6790/0 2024-03-29T11:14:30.614 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.635+0000 7fc92594b640 10 monclient(hunting): picked mon.noname-c con 0x7fc92013ff40 addr v1:172.21.15.70:6789/0 2024-03-29T11:14:30.614 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.635+0000 7fc92594b640 10 monclient(hunting): picked mon.noname-a con 0x7fc9201403f0 addr v1:172.21.15.18:6789/0 2024-03-29T11:14:30.614 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.635+0000 7fc92594b640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9200b4310 con 0x7fc9201403f0 2024-03-29T11:14:30.614 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.635+0000 7fc92594b640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc920140800 con 0x7fc92013ff40 2024-03-29T11:14:30.615 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.635+0000 7fc92594b640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9200aec00 con 0x7fc9200acf60 2024-03-29T11:14:30.615 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.635+0000 7fc92594b640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:30.615 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.635+0000 7fc92594b640 10 monclient(hunting): authenticate will time out at 1096.907394s 2024-03-29T11:14:30.615 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.635+0000 7fc91effd640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fc9200acf60 0x7fc920139b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:35410/0 (socket says 172.21.15.18:35410) 2024-03-29T11:14:30.615 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.635+0000 7fc91effd640 1 -- 172.21.15.18:0/1289179704 learned_addr learned my addr 172.21.15.18:0/1289179704 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:14:30.615 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.635+0000 7fc91dffb640 1 -- 172.21.15.18:0/1289179704 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3542541627 0 0) 0x7fc9200aec00 con 0x7fc9200acf60 2024-03-29T11:14:30.615 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.635+0000 7fc91dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:14:30.615 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.635+0000 7fc91dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:30.615 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.635+0000 7fc91dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:30.615 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.635+0000 7fc91dffb640 10 monclient(hunting): my global_id is 4454 2024-03-29T11:14:30.615 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.635+0000 7fc91dffb640 1 -- 172.21.15.18:0/1289179704 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc908003520 con 0x7fc9200acf60 2024-03-29T11:14:30.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.636+0000 7fc91dffb640 1 -- 172.21.15.18:0/1289179704 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2426065342 0 0) 0x7fc9200b4310 con 0x7fc9201403f0 2024-03-29T11:14:30.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.636+0000 7fc91dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:14:30.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.636+0000 7fc91dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:30.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.636+0000 7fc91dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:30.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.636+0000 7fc91dffb640 10 monclient(hunting): my global_id is 4398 2024-03-29T11:14:30.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.636+0000 7fc91dffb640 1 -- 172.21.15.18:0/1289179704 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9080048a0 con 0x7fc9201403f0 2024-03-29T11:14:30.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.636+0000 7fc91dffb640 1 -- 172.21.15.18:0/1289179704 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4288877995 0 0) 0x7fc9080048a0 con 0x7fc9201403f0 2024-03-29T11:14:30.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.636+0000 7fc91dffb640 1 -- 172.21.15.18:0/1289179704 >> v1:172.21.15.18:6790/0 conn(0x7fc9200acf60 legacy=0x7fc920139b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:30.616 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.636+0000 7fc91dffb640 1 -- 172.21.15.18:0/1289179704 >> v1:172.21.15.70:6789/0 conn(0x7fc92013ff40 legacy=0x7fc920148840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:14:30.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.638+0000 7fc91dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:14:30.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.638+0000 7fc91dffb640 1 monclient: found mon.noname-a 2024-03-29T11:14:30.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.638+0000 7fc91dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:30.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.638+0000 7fc91dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-29T11:14:30.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.638+0000 7fc91dffb640 1 -- 172.21.15.18:0/1289179704 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9200aeee0 con 0x7fc9201403f0 2024-03-29T11:14:30.619 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.638+0000 7fc91dffb640 10 monclient: _finish_auth 0 2024-03-29T11:14:30.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.638+0000 7fc91dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:14:30.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.638+0000 7fc91dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:30.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.638+0000 7fc91dffb640 1 -- 172.21.15.18:0/1289179704 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fc9180019b0 con 0x7fc9201403f0 2024-03-29T11:14:30.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.638+0000 7fc91dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:30.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.638+0000 7fc91dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:14:30.620 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.638+0000 7fc91dffb640 10 monclient: dump: 2024-03-29T11:14:30.620 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:14:30.620 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:30.620 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:30.620 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:30.620 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:30.620 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:30.620 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:14:30.620 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:14:30.620 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:14:30.621 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:30.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.638+0000 7fc92594b640 5 monclient: authenticate success, global_id 4398 2024-03-29T11:14:30.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.638+0000 7fc92594b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:30.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.639+0000 7fc91dffb640 1 -- 172.21.15.18:0/1289179704 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc918001c00 con 0x7fc9201403f0 2024-03-29T11:14:30.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.639+0000 7fc91dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:30.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.639+0000 7fc91dffb640 1 -- 172.21.15.18:0/1289179704 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fc918001f90 con 0x7fc9201403f0 2024-03-29T11:14:30.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.639+0000 7fc91dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:30.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.639+0000 7fc91dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:14:30.621 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.639+0000 7fc91dffb640 10 monclient: dump: 2024-03-29T11:14:30.622 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:14:30.622 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:30.622 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:30.622 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:30.622 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:30.622 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:30.622 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:14:30.622 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:14:30.622 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:14:30.622 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:30.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.639+0000 7fc92594b640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:30.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.639+0000 7fc92594b640 4 set_mon_vals no callback set 2024-03-29T11:14:30.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.640+0000 7fc92594b640 10 monclient: shutdown 2024-03-29T11:14:30.622 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.640+0000 7fc92594b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:30.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.640+0000 7fc92594b640 1 -- 172.21.15.18:0/1289179704 >> v1:172.21.15.18:6789/0 conn(0x7fc9201403f0 legacy=0x7fc92014bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:30.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.641+0000 7fc92594b640 1 -- 172.21.15.18:0/1289179704 shutdown_connections 2024-03-29T11:14:30.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.641+0000 7fc92594b640 1 -- 172.21.15.18:0/1289179704 >> 172.21.15.18:0/1289179704 conn(0x7fc9200b35e0 msgr2=0x7fc9200b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:30.623 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.642+0000 7fc92594b640 1 -- 172.21.15.18:0/1289179704 shutdown_connections 2024-03-29T11:14:30.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.642+0000 7fc92594b640 1 -- 172.21.15.18:0/1289179704 wait complete. 2024-03-29T11:14:30.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.642+0000 7fc92594b640 10 monclient: build_initial_monmap 2024-03-29T11:14:30.628 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.642+0000 7fc92594b640 10 monclient: monmap: 2024-03-29T11:14:30.628 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:14:30.628 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:30.628 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:14:30.628 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:14:30.628 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:30.628 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:30.629 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:14:30.629 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:14:30.629 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:14:30.629 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:30.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.643+0000 7fc92594b640 1 librados: starting msgr at 2024-03-29T11:14:30.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.643+0000 7fc92594b640 1 librados: starting objecter 2024-03-29T11:14:30.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.643+0000 7fc92594b640 1 Processor -- start 2024-03-29T11:14:30.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.643+0000 7fc92594b640 1 -- start start 2024-03-29T11:14:30.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.643+0000 7fc92594b640 1 librados: setting wanted keys 2024-03-29T11:14:30.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.643+0000 7fc92594b640 1 librados: calling monclient init 2024-03-29T11:14:30.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.643+0000 7fc92594b640 10 monclient: init 2024-03-29T11:14:30.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.644+0000 7fc92594b640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:30.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.644+0000 7fc92594b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:14:30.629 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.644+0000 7fc92594b640 10 monclient(hunting): picked mon.noname-c con 0x7fc9200acf60 addr v1:172.21.15.18:6790/0 2024-03-29T11:14:30.630 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.644+0000 7fc92594b640 10 monclient(hunting): picked mon.noname-b con 0x7fc92013ff40 addr v1:172.21.15.70:6789/0 2024-03-29T11:14:30.630 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.644+0000 7fc92594b640 10 monclient(hunting): picked mon.noname-a con 0x7fc9201403f0 addr v1:172.21.15.18:6789/0 2024-03-29T11:14:30.630 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.644+0000 7fc92594b640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc92006f930 con 0x7fc9201403f0 2024-03-29T11:14:30.630 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.644+0000 7fc92594b640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc92013d0a0 con 0x7fc92013ff40 2024-03-29T11:14:30.630 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.644+0000 7fc92594b640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc92013d5c0 con 0x7fc9200acf60 2024-03-29T11:14:30.630 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.644+0000 7fc92594b640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:30.630 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.644+0000 7fc92594b640 10 monclient(hunting): authenticate will time out at 1096.916809s 2024-03-29T11:14:30.630 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.645+0000 7fc91f7fe640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fc9201403f0 0x7fc9201e2b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:39178/0 (socket says 172.21.15.18:39178) 2024-03-29T11:14:30.630 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.645+0000 7fc91f7fe640 1 -- 172.21.15.18:0/2307655076 learned_addr learned my addr 172.21.15.18:0/2307655076 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:14:30.630 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.645+0000 7fc907fff640 10 client.?.objecter ms_handle_connect 0x7fc9201403f0 2024-03-29T11:14:30.630 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.645+0000 7fc907fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:30.630 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.645+0000 7fc907fff640 1 -- 172.21.15.18:0/2307655076 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 412891051 0 0) 0x7fc92006f930 con 0x7fc9201403f0 2024-03-29T11:14:30.630 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.645+0000 7fc907fff640 10 monclient(hunting): _negotiate 2024-03-29T11:14:30.630 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.645+0000 7fc907fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:30.631 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.645+0000 7fc907fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:30.631 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.645+0000 7fc907fff640 10 monclient(hunting): my global_id is 4401 2024-03-29T11:14:30.631 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.645+0000 7fc907fff640 1 -- 172.21.15.18:0/2307655076 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc90c004eb0 con 0x7fc9201403f0 2024-03-29T11:14:30.631 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.646+0000 7fc907fff640 1 -- 172.21.15.18:0/2307655076 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3619037736 0 0) 0x7fc90c004eb0 con 0x7fc9201403f0 2024-03-29T11:14:30.631 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.646+0000 7fc907fff640 1 -- 172.21.15.18:0/2307655076 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc92006f930 con 0x7fc9201403f0 2024-03-29T11:14:30.631 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.646+0000 7fc907fff640 1 -- 172.21.15.18:0/2307655076 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fc918001fd0 con 0x7fc9201403f0 2024-03-29T11:14:30.631 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.646+0000 7fc907fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:30.631 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.646+0000 7fc907fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:14:30.631 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.646+0000 7fc907fff640 10 monclient(hunting): dump: 2024-03-29T11:14:30.631 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:14:30.631 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:30.631 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:30.631 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:30.632 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:30.632 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:30.632 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:14:30.632 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:14:30.632 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:14:30.632 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:30.632 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.646+0000 7fc907fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:30.632 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.646+0000 7fc907fff640 10 client.?.objecter ms_handle_connect 0x7fc9200acf60 2024-03-29T11:14:30.632 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.646+0000 7fc907fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:30.632 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.646+0000 7fc907fff640 1 -- 172.21.15.18:0/2307655076 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1963315362 0 0) 0x7fc92013d5c0 con 0x7fc9200acf60 2024-03-29T11:14:30.633 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.646+0000 7fc907fff640 10 monclient(hunting): _negotiate 2024-03-29T11:14:30.633 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.646+0000 7fc907fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:30.633 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.646+0000 7fc907fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:30.633 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.646+0000 7fc907fff640 10 monclient(hunting): my global_id is 4457 2024-03-29T11:14:30.633 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.646+0000 7fc907fff640 1 -- 172.21.15.18:0/2307655076 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc90c004eb0 con 0x7fc9200acf60 2024-03-29T11:14:30.633 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.646+0000 7fc907fff640 1 -- 172.21.15.18:0/2307655076 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4068159085 0 0) 0x7fc92006f930 con 0x7fc9201403f0 2024-03-29T11:14:30.633 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.646+0000 7fc907fff640 1 -- 172.21.15.18:0/2307655076 >> v1:172.21.15.18:6790/0 conn(0x7fc9200acf60 legacy=0x7fc920133f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:30.633 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.646+0000 7fc907fff640 1 -- 172.21.15.18:0/2307655076 >> v1:172.21.15.70:6789/0 conn(0x7fc92013ff40 legacy=0x7fc9201da850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:14:30.633 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.648+0000 7fc907fff640 10 monclient: _finish_hunting 0 2024-03-29T11:14:30.633 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.648+0000 7fc907fff640 1 monclient: found mon.a 2024-03-29T11:14:30.633 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.648+0000 7fc907fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:30.633 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.648+0000 7fc907fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:14:30.633 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.648+0000 7fc907fff640 1 -- 172.21.15.18:0/2307655076 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc92013c270 con 0x7fc9201403f0 2024-03-29T11:14:30.633 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.648+0000 7fc907fff640 10 monclient: _finish_auth 0 2024-03-29T11:14:30.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.648+0000 7fc907fff640 10 monclient: _check_auth_tickets 2024-03-29T11:14:30.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.648+0000 7fc907fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:30.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.649+0000 7fc92594b640 5 monclient: authenticate success, global_id 4401 2024-03-29T11:14:30.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.649+0000 7fc92594b640 10 monclient: _renew_subs 2024-03-29T11:14:30.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.649+0000 7fc92594b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:14:30.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.649+0000 7fc92594b640 1 -- 172.21.15.18:0/2307655076 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc92013cb80 con 0x7fc9201403f0 2024-03-29T11:14:30.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.649+0000 7fc92594b640 10 client.4401.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:30.634 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.649+0000 7fc92594b640 10 monclient: _renew_subs 2024-03-29T11:14:30.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.649+0000 7fc92594b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:14:30.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.649+0000 7fc92594b640 1 -- 172.21.15.18:0/2307655076 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc920134810 con 0x7fc9201403f0 2024-03-29T11:14:30.635 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.649+0000 7fc92594b640 1 librados: init done 2024-03-29T11:14:30.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.650+0000 7fc907fff640 1 -- 172.21.15.18:0/2307655076 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc918003c20 con 0x7fc9201403f0 2024-03-29T11:14:30.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.650+0000 7fc907fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:30.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.650+0000 7fc91d7fa640 4 set_mon_vals no callback set 2024-03-29T11:14:30.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.650+0000 7fc92594b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:30.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.650+0000 7fc907fff640 1 -- 172.21.15.18:0/2307655076 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fc918004030 con 0x7fc9201403f0 2024-03-29T11:14:30.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.650+0000 7fc907fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:30.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.651+0000 7fc907fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:14:30.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.651+0000 7fc907fff640 10 monclient: dump: 2024-03-29T11:14:30.637 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:14:30.638 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:30.638 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:30.638 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:30.638 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:30.638 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:30.638 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:14:30.638 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:14:30.638 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:14:30.638 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:30.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.652+0000 7fc92594b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:30.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.652+0000 7fc92594b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:14:30.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.652+0000 7fc92594b640 1 -- 172.21.15.18:0/2307655076 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc900005080 con 0x7fc9201403f0 2024-03-29T11:14:30.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.654+0000 7fc907fff640 1 -- 172.21.15.18:0/2307655076 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7fc918004190 con 0x7fc9201403f0 2024-03-29T11:14:30.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.654+0000 7fc91effd640 10 monclient: get_auth_request con 0x7fc90c06dcd0 auth_method 0 2024-03-29T11:14:30.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.654+0000 7fc907fff640 1 -- 172.21.15.18:0/2307655076 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7fc918084f90 con 0x7fc9201403f0 2024-03-29T11:14:30.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.654+0000 7fc907fff640 10 client.4401.objecter ms_dispatch 0x7fc920000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:30.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.654+0000 7fc907fff640 3 client.4401.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:30.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.654+0000 7fc907fff640 3 client.4401.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:30.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.655+0000 7fc907fff640 20 client.4401.objecter dump_active .. 0 homeless 2024-03-29T11:14:30.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.655+0000 7fc907fff640 10 client.4401.objecter ms_handle_connect 0x7fc90c06dcd0 2024-03-29T11:14:30.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.662+0000 7fc907fff640 1 -- 172.21.15.18:0/2307655076 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc918050870 con 0x7fc9201403f0 2024-03-29T11:14:30.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.662+0000 7fc907fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:30.640 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.662+0000 7fc907fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:30.957 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.979+0000 7fc92594b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:30.957 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.979+0000 7fc92594b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:30.957 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.979+0000 7fc92594b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:14:30.957 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.979+0000 7fc92594b640 1 -- 172.21.15.18:0/2307655076 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc900005330 con 0x7fc9201403f0 2024-03-29T11:14:30.958 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.980+0000 7fc907fff640 1 -- 172.21.15.18:0/2307655076 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7fc9180527a0 con 0x7fc9201403f0 2024-03-29T11:14:30.958 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.980+0000 7fc907fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:30.958 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.980+0000 7fc907fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:30.958 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:14:30.959 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:30.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.984+0000 7fc92594b640 10 librados: watch_flush enter 2024-03-29T11:14:30.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.984+0000 7fc92594b640 10 librados: watch_flush exit 2024-03-29T11:14:30.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.984+0000 7fc92594b640 20 client.4401.objecter shutdown clearing up homeless session... 2024-03-29T11:14:30.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.984+0000 7fc92594b640 10 client.4401.objecter successfully canceled tick 2024-03-29T11:14:30.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.984+0000 7fc92594b640 1 -- 172.21.15.18:0/2307655076 >> v1:172.21.15.70:6816/3101871562 conn(0x7fc90c06dcd0 legacy=0x7fc90c070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:30.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.984+0000 7fc92594b640 10 monclient: shutdown 2024-03-29T11:14:30.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.984+0000 7fc92594b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:30.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.984+0000 7fc92594b640 1 -- 172.21.15.18:0/2307655076 >> v1:172.21.15.18:6789/0 conn(0x7fc9201403f0 legacy=0x7fc9201e2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:30.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.985+0000 7fc92594b640 1 -- 172.21.15.18:0/2307655076 shutdown_connections 2024-03-29T11:14:30.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.985+0000 7fc92594b640 1 -- 172.21.15.18:0/2307655076 >> 172.21.15.18:0/2307655076 conn(0x7fc9200b35e0 msgr2=0x7fc9200b1fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:30.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.985+0000 7fc92594b640 1 -- 172.21.15.18:0/2307655076 shutdown_connections 2024-03-29T11:14:30.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.985+0000 7fc92594b640 1 -- 172.21.15.18:0/2307655076 wait complete. 2024-03-29T11:14:30.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:30.985+0000 7fc92594b640 1 librados: shutdown 2024-03-29T11:14:30.979 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:37.789 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:37.941 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.964+0000 7f995cf15640 10 monclient: get_monmap_and_config 2024-03-29T11:14:37.942 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.964+0000 7f995cf15640 10 monclient: build_initial_monmap 2024-03-29T11:14:37.942 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.964+0000 7f995cf15640 10 monclient: monmap: 2024-03-29T11:14:37.942 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:14:37.942 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:37.942 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:14:37.965093+0000 2024-03-29T11:14:37.942 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:14:37.965093+0000 2024-03-29T11:14:37.942 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:37.942 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:37.942 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:14:37.942 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:14:37.942 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:14:37.942 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:37.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.965+0000 7f995cf15640 1 Processor -- start 2024-03-29T11:14:37.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.965+0000 7f995cf15640 1 -- start start 2024-03-29T11:14:37.943 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.965+0000 7f995cf15640 10 monclient: init 2024-03-29T11:14:37.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.966+0000 7f995cf15640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:37.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.966+0000 7f995cf15640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:14:37.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.966+0000 7f995cf15640 10 monclient(hunting): picked mon.noname-b con 0x7f99581457f0 addr v1:172.21.15.18:6790/0 2024-03-29T11:14:37.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.966+0000 7f995cf15640 10 monclient(hunting): picked mon.noname-a con 0x7f995813f650 addr v1:172.21.15.18:6789/0 2024-03-29T11:14:37.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.966+0000 7f995cf15640 10 monclient(hunting): picked mon.noname-c con 0x7f99580a9d70 addr v1:172.21.15.70:6789/0 2024-03-29T11:14:37.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.966+0000 7f995cf15640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99580aaef0 con 0x7f995813f650 2024-03-29T11:14:37.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.966+0000 7f995cf15640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9958145bd0 con 0x7f99580a9d70 2024-03-29T11:14:37.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.966+0000 7f995cf15640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9958145eb0 con 0x7f99581457f0 2024-03-29T11:14:37.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.966+0000 7f995cf15640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:37.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.966+0000 7f995cf15640 10 monclient(hunting): authenticate will time out at 1104.238236s 2024-03-29T11:14:37.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.966+0000 7f9955d9b640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f995813f650 0x7f99580b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:49212/0 (socket says 172.21.15.18:49212) 2024-03-29T11:14:37.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.966+0000 7f9955d9b640 1 -- 172.21.15.18:0/1384001603 learned_addr learned my addr 172.21.15.18:0/1384001603 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:14:37.946 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.967+0000 7f995559a640 1 -- 172.21.15.18:0/1384001603 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3805147814 0 0) 0x7f9958145bd0 con 0x7f99580a9d70 2024-03-29T11:14:37.946 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.967+0000 7f995559a640 10 monclient(hunting): _negotiate 2024-03-29T11:14:37.946 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.967+0000 7f995559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:37.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.967+0000 7f995559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:37.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.967+0000 7f995559a640 10 monclient(hunting): my global_id is 4390 2024-03-29T11:14:37.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.967+0000 7f995559a640 1 -- 172.21.15.18:0/1384001603 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9940003520 con 0x7f99580a9d70 2024-03-29T11:14:37.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.967+0000 7f995559a640 1 -- 172.21.15.18:0/1384001603 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 628267524 0 0) 0x7f99580aaef0 con 0x7f995813f650 2024-03-29T11:14:37.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.967+0000 7f995559a640 10 monclient(hunting): _negotiate 2024-03-29T11:14:37.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.967+0000 7f995559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:37.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.967+0000 7f995559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:37.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.967+0000 7f995559a640 10 monclient(hunting): my global_id is 4404 2024-03-29T11:14:37.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.967+0000 7f995559a640 1 -- 172.21.15.18:0/1384001603 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9958145bd0 con 0x7f995813f650 2024-03-29T11:14:37.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.967+0000 7f995559a640 1 -- 172.21.15.18:0/1384001603 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1810791821 0 0) 0x7f9958145eb0 con 0x7f99581457f0 2024-03-29T11:14:37.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.967+0000 7f995559a640 10 monclient(hunting): _negotiate 2024-03-29T11:14:37.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.967+0000 7f995559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:37.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.967+0000 7f995559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:37.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.967+0000 7f995559a640 10 monclient(hunting): my global_id is 4460 2024-03-29T11:14:37.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.967+0000 7f995559a640 1 -- 172.21.15.18:0/1384001603 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99580aaef0 con 0x7f99581457f0 2024-03-29T11:14:37.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.967+0000 7f995559a640 1 -- 172.21.15.18:0/1384001603 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2309864273 0 0) 0x7f9940003520 con 0x7f99580a9d70 2024-03-29T11:14:37.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.968+0000 7f995559a640 1 -- 172.21.15.18:0/1384001603 >> v1:172.21.15.18:6790/0 conn(0x7f99581457f0 legacy=0x7f995813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:37.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.968+0000 7f995559a640 1 -- 172.21.15.18:0/1384001603 >> v1:172.21.15.18:6789/0 conn(0x7f995813f650 legacy=0x7f99580b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:37.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.968+0000 7f995559a640 10 monclient: _finish_hunting 0 2024-03-29T11:14:37.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.968+0000 7f995559a640 1 monclient: found mon.noname-c 2024-03-29T11:14:37.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.968+0000 7f995559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:37.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.968+0000 7f995559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.70:6789/0 2024-03-29T11:14:37.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.968+0000 7f995559a640 1 -- 172.21.15.18:0/1384001603 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9958146340 con 0x7f99580a9d70 2024-03-29T11:14:37.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.968+0000 7f995559a640 10 monclient: _finish_auth 0 2024-03-29T11:14:37.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.968+0000 7f995559a640 10 monclient: _check_auth_tickets 2024-03-29T11:14:37.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.968+0000 7f995559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:37.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.968+0000 7f995cf15640 5 monclient: authenticate success, global_id 4390 2024-03-29T11:14:37.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.968+0000 7f995559a640 1 -- 172.21.15.18:0/1384001603 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f9950001940 con 0x7f99580a9d70 2024-03-29T11:14:37.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.968+0000 7f995cf15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:37.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.968+0000 7f995559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:37.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.968+0000 7f995559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:14:37.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.968+0000 7f995559a640 10 monclient: dump: 2024-03-29T11:14:37.949 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:14:37.949 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:37.949 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:37.949 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:37.949 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:37.949 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:37.949 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:14:37.949 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:14:37.949 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:14:37.949 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:37.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.968+0000 7f995cf15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:37.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.968+0000 7f995559a640 1 -- 172.21.15.18:0/1384001603 <== mon.1 v1:172.21.15.70:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9950001b60 con 0x7f99580a9d70 2024-03-29T11:14:37.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.968+0000 7f995559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:37.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.968+0000 7f995559a640 1 -- 172.21.15.18:0/1384001603 <== mon.1 v1:172.21.15.70:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f9950001ef0 con 0x7f99580a9d70 2024-03-29T11:14:37.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.968+0000 7f995559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:37.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.968+0000 7f995559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:37.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.968+0000 7f995559a640 10 monclient: dump: 2024-03-29T11:14:37.950 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:14:37.950 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:37.950 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:37.950 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:37.950 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:37.950 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:37.951 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:14:37.951 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:14:37.951 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:14:37.951 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:37.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.968+0000 7f995cf15640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:37.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.968+0000 7f995cf15640 4 set_mon_vals no callback set 2024-03-29T11:14:37.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.969+0000 7f995cf15640 10 monclient: shutdown 2024-03-29T11:14:37.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.969+0000 7f995cf15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:37.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.969+0000 7f995cf15640 1 -- 172.21.15.18:0/1384001603 >> v1:172.21.15.70:6789/0 conn(0x7f99580a9d70 legacy=0x7f995814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:37.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.970+0000 7f995cf15640 1 -- 172.21.15.18:0/1384001603 shutdown_connections 2024-03-29T11:14:37.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.970+0000 7f995cf15640 1 -- 172.21.15.18:0/1384001603 >> 172.21.15.18:0/1384001603 conn(0x7f9958066a00 msgr2=0x7f9958066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:37.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.970+0000 7f995cf15640 1 -- 172.21.15.18:0/1384001603 shutdown_connections 2024-03-29T11:14:37.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.970+0000 7f995cf15640 1 -- 172.21.15.18:0/1384001603 wait complete. 2024-03-29T11:14:37.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.971+0000 7f995cf15640 10 monclient: build_initial_monmap 2024-03-29T11:14:37.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.971+0000 7f995cf15640 10 monclient: monmap: 2024-03-29T11:14:37.962 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:14:37.962 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:37.962 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:14:37.962 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:14:37.962 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:37.962 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:37.962 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:14:37.962 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:14:37.962 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:14:37.962 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:37.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.972+0000 7f995cf15640 1 librados: starting msgr at 2024-03-29T11:14:37.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.972+0000 7f995cf15640 1 librados: starting objecter 2024-03-29T11:14:37.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.972+0000 7f995cf15640 1 Processor -- start 2024-03-29T11:14:37.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.972+0000 7f995cf15640 1 -- start start 2024-03-29T11:14:37.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.972+0000 7f995cf15640 1 librados: setting wanted keys 2024-03-29T11:14:37.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.972+0000 7f995cf15640 1 librados: calling monclient init 2024-03-29T11:14:37.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.972+0000 7f995cf15640 10 monclient: init 2024-03-29T11:14:37.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.973+0000 7f995cf15640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:37.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.973+0000 7f995cf15640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:14:37.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.973+0000 7f995cf15640 10 monclient(hunting): picked mon.noname-b con 0x7f99580a9d70 addr v1:172.21.15.70:6789/0 2024-03-29T11:14:37.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.973+0000 7f995cf15640 10 monclient(hunting): picked mon.noname-a con 0x7f995813f650 addr v1:172.21.15.18:6789/0 2024-03-29T11:14:37.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.973+0000 7f995cf15640 10 monclient(hunting): picked mon.noname-c con 0x7f99581457f0 addr v1:172.21.15.18:6790/0 2024-03-29T11:14:37.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.973+0000 7f995cf15640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f995813b050 con 0x7f995813f650 2024-03-29T11:14:37.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.973+0000 7f995cf15640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9958142710 con 0x7f99580a9d70 2024-03-29T11:14:37.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.973+0000 7f995cf15640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f995806ed20 con 0x7f99581457f0 2024-03-29T11:14:37.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.973+0000 7f995cf15640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:37.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.973+0000 7f995cf15640 10 monclient(hunting): authenticate will time out at 1104.245323s 2024-03-29T11:14:37.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.973+0000 7f9956d9d640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f99581457f0 0x7f995813b4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:43694/0 (socket says 172.21.15.18:43694) 2024-03-29T11:14:37.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.973+0000 7f9955d9b640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f995813f650 0x7f9958142000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:49514/0 (socket says 172.21.15.18:49514) 2024-03-29T11:14:37.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.973+0000 7f9955d9b640 1 -- 172.21.15.18:0/1903618632 learned_addr learned my addr 172.21.15.18:0/1903618632 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:14:37.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.974+0000 7f9946ffd640 10 client.?.objecter ms_handle_connect 0x7f995813f650 2024-03-29T11:14:37.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.974+0000 7f9946ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:37.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.974+0000 7f9946ffd640 10 client.?.objecter ms_handle_connect 0x7f99580a9d70 2024-03-29T11:14:37.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.974+0000 7f9946ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:37.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.975+0000 7f9946ffd640 1 -- 172.21.15.18:0/1903618632 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3838004693 0 0) 0x7f9958142710 con 0x7f99580a9d70 2024-03-29T11:14:37.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.975+0000 7f9946ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:37.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.975+0000 7f9946ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:37.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.975+0000 7f9946ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:37.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.975+0000 7f9946ffd640 10 monclient(hunting): my global_id is 4393 2024-03-29T11:14:37.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.975+0000 7f9946ffd640 1 -- 172.21.15.18:0/1903618632 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9938004eb0 con 0x7f99580a9d70 2024-03-29T11:14:37.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.975+0000 7f9946ffd640 10 client.?.objecter ms_handle_connect 0x7f99581457f0 2024-03-29T11:14:37.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.975+0000 7f9946ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:37.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.975+0000 7f9946ffd640 1 -- 172.21.15.18:0/1903618632 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3601346090 0 0) 0x7f995813b050 con 0x7f995813f650 2024-03-29T11:14:37.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.975+0000 7f9946ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:37.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.975+0000 7f9946ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:37.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.975+0000 7f9946ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:37.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.975+0000 7f9946ffd640 10 monclient(hunting): my global_id is 4407 2024-03-29T11:14:37.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.975+0000 7f9946ffd640 1 -- 172.21.15.18:0/1903618632 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9958142710 con 0x7f995813f650 2024-03-29T11:14:37.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.975+0000 7f9946ffd640 1 -- 172.21.15.18:0/1903618632 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1483992472 0 0) 0x7f9938004eb0 con 0x7f99580a9d70 2024-03-29T11:14:37.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.975+0000 7f9946ffd640 1 -- 172.21.15.18:0/1903618632 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f995813b050 con 0x7f99580a9d70 2024-03-29T11:14:37.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.975+0000 7f9946ffd640 1 -- 172.21.15.18:0/1903618632 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f994c001800 con 0x7f99580a9d70 2024-03-29T11:14:37.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.975+0000 7f9946ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:37.966 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.975+0000 7f9946ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:37.966 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.975+0000 7f9946ffd640 10 monclient(hunting): dump: 2024-03-29T11:14:37.966 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:14:37.966 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:37.966 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:37.966 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:37.966 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:37.966 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:37.966 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:14:37.966 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:14:37.966 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:14:37.966 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:37.966 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.975+0000 7f9946ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:37.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.975+0000 7f9946ffd640 1 -- 172.21.15.18:0/1903618632 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1881838526 0 0) 0x7f995806ed20 con 0x7f99581457f0 2024-03-29T11:14:37.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.975+0000 7f9946ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:14:37.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.975+0000 7f9946ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:37.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.975+0000 7f9946ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:37.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.975+0000 7f9946ffd640 10 monclient(hunting): my global_id is 4463 2024-03-29T11:14:37.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.975+0000 7f9946ffd640 1 -- 172.21.15.18:0/1903618632 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9938004eb0 con 0x7f99581457f0 2024-03-29T11:14:37.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.976+0000 7f9946ffd640 1 -- 172.21.15.18:0/1903618632 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 491008080 0 0) 0x7f9958142710 con 0x7f995813f650 2024-03-29T11:14:37.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.976+0000 7f9946ffd640 1 -- 172.21.15.18:0/1903618632 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f995806ed20 con 0x7f995813f650 2024-03-29T11:14:37.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.976+0000 7f9946ffd640 1 -- 172.21.15.18:0/1903618632 <== mon.1 v1:172.21.15.70:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3625505950 0 0) 0x7f995813b050 con 0x7f99580a9d70 2024-03-29T11:14:37.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.976+0000 7f9946ffd640 1 -- 172.21.15.18:0/1903618632 >> v1:172.21.15.18:6790/0 conn(0x7f99581457f0 legacy=0x7f995813b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:37.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.976+0000 7f9946ffd640 1 -- 172.21.15.18:0/1903618632 >> v1:172.21.15.18:6789/0 conn(0x7f995813f650 legacy=0x7f9958142000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:37.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.976+0000 7f9946ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:14:37.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.976+0000 7f9946ffd640 1 monclient: found mon.b 2024-03-29T11:14:37.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.976+0000 7f9946ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:37.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.976+0000 7f9946ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:14:37.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.976+0000 7f9946ffd640 1 -- 172.21.15.18:0/1903618632 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f995813bbf0 con 0x7f99580a9d70 2024-03-29T11:14:37.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.976+0000 7f9946ffd640 10 monclient: _finish_auth 0 2024-03-29T11:14:37.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.976+0000 7f9946ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:14:37.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.976+0000 7f9946ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:37.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.976+0000 7f995cf15640 5 monclient: authenticate success, global_id 4393 2024-03-29T11:14:37.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.976+0000 7f995cf15640 10 monclient: _renew_subs 2024-03-29T11:14:37.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.976+0000 7f995cf15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:14:37.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.976+0000 7f995cf15640 1 -- 172.21.15.18:0/1903618632 --> v1:172.21.15.70:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9958139d60 con 0x7f99580a9d70 2024-03-29T11:14:37.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.976+0000 7f995cf15640 10 client.4393.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:37.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.976+0000 7f995cf15640 10 monclient: _renew_subs 2024-03-29T11:14:37.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.976+0000 7f995cf15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:14:37.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.976+0000 7f995cf15640 1 -- 172.21.15.18:0/1903618632 --> v1:172.21.15.70:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9958138ea0 con 0x7f99580a9d70 2024-03-29T11:14:37.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.976+0000 7f995cf15640 1 librados: init done 2024-03-29T11:14:37.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.978+0000 7f9946ffd640 1 -- 172.21.15.18:0/1903618632 <== mon.1 v1:172.21.15.70:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f994c001d10 con 0x7f99580a9d70 2024-03-29T11:14:37.969 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.978+0000 7f9946ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:37.969 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.978+0000 7f9946ffd640 1 -- 172.21.15.18:0/1903618632 <== mon.1 v1:172.21.15.70:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f994c0037b0 con 0x7f99580a9d70 2024-03-29T11:14:37.969 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.978+0000 7f9946ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:37.969 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.978+0000 7f9954d99640 4 set_mon_vals no callback set 2024-03-29T11:14:37.969 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.978+0000 7f9946ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:14:37.969 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.978+0000 7f9946ffd640 10 monclient: dump: 2024-03-29T11:14:37.969 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:14:37.969 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:37.969 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:37.969 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:37.969 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:37.969 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:37.969 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:14:37.969 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:14:37.970 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:14:37.970 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:37.970 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.978+0000 7f9944ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:37.970 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.979+0000 7f9944ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:37.970 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.979+0000 7f9944ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:14:37.970 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.979+0000 7f9944ff9640 1 -- 172.21.15.18:0/1903618632 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9958075f10 con 0x7f99580a9d70 2024-03-29T11:14:37.970 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.980+0000 7f9946ffd640 1 -- 172.21.15.18:0/1903618632 <== mon.1 v1:172.21.15.70:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f994c003990 con 0x7f99580a9d70 2024-03-29T11:14:37.970 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.981+0000 7f9946ffd640 1 -- 172.21.15.18:0/1903618632 <== mon.1 v1:172.21.15.70:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f994c083640 con 0x7f99580a9d70 2024-03-29T11:14:37.970 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.981+0000 7f9946ffd640 10 client.4393.objecter ms_dispatch 0x7f9958000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:37.970 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.981+0000 7f9946ffd640 3 client.4393.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:37.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.981+0000 7f9946ffd640 3 client.4393.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:37.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.982+0000 7f9946ffd640 20 client.4393.objecter dump_active .. 0 homeless 2024-03-29T11:14:37.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.982+0000 7f9955d9b640 10 monclient: get_auth_request con 0x7f993806dea0 auth_method 0 2024-03-29T11:14:37.971 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.982+0000 7f9946ffd640 10 client.4393.objecter ms_handle_connect 0x7f993806dea0 2024-03-29T11:14:37.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.988+0000 7f9946ffd640 1 -- 172.21.15.18:0/1903618632 <== mon.1 v1:172.21.15.70:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f994c04f060 con 0x7f99580a9d70 2024-03-29T11:14:37.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.988+0000 7f9946ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:37.972 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:37.988+0000 7f9946ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:38.331 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:38.353+0000 7f9944ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:38.332 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:38.353+0000 7f9944ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:38.332 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:38.353+0000 7f9944ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:14:38.332 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:38.353+0000 7f9944ff9640 1 -- 172.21.15.18:0/1903618632 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f995804ef70 con 0x7f99580a9d70 2024-03-29T11:14:38.332 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:38.354+0000 7f9946ffd640 1 -- 172.21.15.18:0/1903618632 <== mon.1 v1:172.21.15.70:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f994c050f90 con 0x7f99580a9d70 2024-03-29T11:14:38.332 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:38.354+0000 7f9946ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:38.332 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:38.354+0000 7f9946ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:38.333 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:14:38.333 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:38.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:38.357+0000 7f9944ff9640 10 librados: watch_flush enter 2024-03-29T11:14:38.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:38.357+0000 7f9944ff9640 10 librados: watch_flush exit 2024-03-29T11:14:38.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:38.357+0000 7f9944ff9640 20 client.4393.objecter shutdown clearing up homeless session... 2024-03-29T11:14:38.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:38.357+0000 7f9944ff9640 10 client.4393.objecter successfully canceled tick 2024-03-29T11:14:38.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:38.358+0000 7f9944ff9640 1 -- 172.21.15.18:0/1903618632 >> v1:172.21.15.70:6816/3101871562 conn(0x7f993806dea0 legacy=0x7f9938070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:38.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:38.358+0000 7f9944ff9640 10 monclient: shutdown 2024-03-29T11:14:38.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:38.358+0000 7f9944ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:38.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:38.358+0000 7f9944ff9640 1 -- 172.21.15.18:0/1903618632 >> v1:172.21.15.70:6789/0 conn(0x7f99580a9d70 legacy=0x7f995813a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:38.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:38.358+0000 7f9944ff9640 1 -- 172.21.15.18:0/1903618632 shutdown_connections 2024-03-29T11:14:38.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:38.358+0000 7f9944ff9640 1 -- 172.21.15.18:0/1903618632 >> 172.21.15.18:0/1903618632 conn(0x7f9958066a00 msgr2=0x7f99581d2420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:38.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:38.358+0000 7f9944ff9640 1 -- 172.21.15.18:0/1903618632 shutdown_connections 2024-03-29T11:14:38.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:38.358+0000 7f9944ff9640 1 -- 172.21.15.18:0/1903618632 wait complete. 2024-03-29T11:14:38.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:38.358+0000 7f9944ff9640 1 librados: shutdown 2024-03-29T11:14:38.350 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:45.160 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:45.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.332+0000 7ff263684640 10 monclient: get_monmap_and_config 2024-03-29T11:14:45.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.332+0000 7ff263684640 10 monclient: build_initial_monmap 2024-03-29T11:14:45.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.332+0000 7ff263684640 10 monclient: monmap: 2024-03-29T11:14:45.310 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:14:45.310 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:45.310 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:14:45.333428+0000 2024-03-29T11:14:45.310 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:14:45.333428+0000 2024-03-29T11:14:45.311 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:45.311 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:45.311 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:14:45.311 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:14:45.311 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:14:45.311 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:45.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.333+0000 7ff263684640 1 Processor -- start 2024-03-29T11:14:45.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.333+0000 7ff263684640 1 -- start start 2024-03-29T11:14:45.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.333+0000 7ff263684640 10 monclient: init 2024-03-29T11:14:45.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.334+0000 7ff263684640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:45.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.334+0000 7ff263684640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:14:45.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.334+0000 7ff263684640 10 monclient(hunting): picked mon.noname-b con 0x7ff25c1457f0 addr v1:172.21.15.18:6790/0 2024-03-29T11:14:45.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.334+0000 7ff263684640 10 monclient(hunting): picked mon.noname-a con 0x7ff25c13f650 addr v1:172.21.15.18:6789/0 2024-03-29T11:14:45.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.334+0000 7ff263684640 10 monclient(hunting): picked mon.noname-c con 0x7ff25c0a9d70 addr v1:172.21.15.70:6789/0 2024-03-29T11:14:45.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.334+0000 7ff263684640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff25c0aaef0 con 0x7ff25c13f650 2024-03-29T11:14:45.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.334+0000 7ff263684640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff25c145bd0 con 0x7ff25c0a9d70 2024-03-29T11:14:45.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.334+0000 7ff263684640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff25c145eb0 con 0x7ff25c1457f0 2024-03-29T11:14:45.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.334+0000 7ff263684640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:45.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.334+0000 7ff263684640 10 monclient(hunting): authenticate will time out at 1111.606537s 2024-03-29T11:14:45.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.337+0000 7ff260c1f640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7ff25c13f650 0x7ff25c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:60318/0 (socket says 172.21.15.18:60318) 2024-03-29T11:14:45.315 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.337+0000 7ff260c1f640 1 -- 172.21.15.18:0/2892878558 learned_addr learned my addr 172.21.15.18:0/2892878558 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:14:45.316 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.337+0000 7ff253fff640 1 -- 172.21.15.18:0/2892878558 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3228853355 0 0) 0x7ff25c0aaef0 con 0x7ff25c13f650 2024-03-29T11:14:45.316 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.337+0000 7ff253fff640 10 monclient(hunting): _negotiate 2024-03-29T11:14:45.316 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.337+0000 7ff253fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:45.316 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.337+0000 7ff253fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:45.316 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.337+0000 7ff253fff640 10 monclient(hunting): my global_id is 4410 2024-03-29T11:14:45.316 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.337+0000 7ff253fff640 1 -- 172.21.15.18:0/2892878558 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff248003520 con 0x7ff25c13f650 2024-03-29T11:14:45.316 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.338+0000 7ff253fff640 1 -- 172.21.15.18:0/2892878558 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2186447999 0 0) 0x7ff248003520 con 0x7ff25c13f650 2024-03-29T11:14:45.316 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.338+0000 7ff253fff640 1 -- 172.21.15.18:0/2892878558 >> v1:172.21.15.18:6790/0 conn(0x7ff25c1457f0 legacy=0x7ff25c13ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:14:45.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.339+0000 7ff253fff640 1 -- 172.21.15.18:0/2892878558 >> v1:172.21.15.70:6789/0 conn(0x7ff25c0a9d70 legacy=0x7ff25c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:14:45.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.339+0000 7ff253fff640 10 monclient: _finish_hunting 0 2024-03-29T11:14:45.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.339+0000 7ff253fff640 1 monclient: found mon.noname-a 2024-03-29T11:14:45.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.339+0000 7ff253fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:45.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.339+0000 7ff253fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-29T11:14:45.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.339+0000 7ff253fff640 1 -- 172.21.15.18:0/2892878558 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff25c146340 con 0x7ff25c13f650 2024-03-29T11:14:45.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.339+0000 7ff253fff640 10 monclient: _finish_auth 0 2024-03-29T11:14:45.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.339+0000 7ff253fff640 10 monclient: _check_auth_tickets 2024-03-29T11:14:45.319 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.339+0000 7ff253fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:45.319 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.339+0000 7ff253fff640 1 -- 172.21.15.18:0/2892878558 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7ff258001670 con 0x7ff25c13f650 2024-03-29T11:14:45.319 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.339+0000 7ff253fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:45.319 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.339+0000 7ff253fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:14:45.319 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.339+0000 7ff253fff640 10 monclient: dump: 2024-03-29T11:14:45.319 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:14:45.319 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:45.319 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:45.319 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:45.319 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:45.319 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:45.319 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:14:45.319 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:14:45.319 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:14:45.320 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:45.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.339+0000 7ff263684640 5 monclient: authenticate success, global_id 4410 2024-03-29T11:14:45.320 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.339+0000 7ff263684640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:14:45.321 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.342+0000 7ff253fff640 1 -- 172.21.15.18:0/2892878558 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2580018d0 con 0x7ff25c13f650 2024-03-29T11:14:45.321 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.342+0000 7ff253fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:45.321 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.342+0000 7ff263684640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:45.321 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.342+0000 7ff263684640 4 set_mon_vals no callback set 2024-03-29T11:14:45.321 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.342+0000 7ff253fff640 1 -- 172.21.15.18:0/2892878558 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7ff258001d20 con 0x7ff25c13f650 2024-03-29T11:14:45.321 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.342+0000 7ff253fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:45.321 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.342+0000 7ff253fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:14:45.321 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.342+0000 7ff253fff640 10 monclient: dump: 2024-03-29T11:14:45.321 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:14:45.321 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:45.321 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:45.321 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:45.321 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:45.322 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:45.322 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:14:45.322 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:14:45.322 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:14:45.322 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:45.322 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.343+0000 7ff263684640 10 monclient: shutdown 2024-03-29T11:14:45.322 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.343+0000 7ff263684640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:45.322 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.343+0000 7ff263684640 1 -- 172.21.15.18:0/2892878558 >> v1:172.21.15.18:6789/0 conn(0x7ff25c13f650 legacy=0x7ff25c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:45.322 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.343+0000 7ff263684640 1 -- 172.21.15.18:0/2892878558 shutdown_connections 2024-03-29T11:14:45.322 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.343+0000 7ff263684640 1 -- 172.21.15.18:0/2892878558 >> 172.21.15.18:0/2892878558 conn(0x7ff25c066a00 msgr2=0x7ff25c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:45.323 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.345+0000 7ff263684640 1 -- 172.21.15.18:0/2892878558 shutdown_connections 2024-03-29T11:14:45.323 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.345+0000 7ff263684640 1 -- 172.21.15.18:0/2892878558 wait complete. 2024-03-29T11:14:45.324 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.346+0000 7ff263684640 10 monclient: build_initial_monmap 2024-03-29T11:14:45.324 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.346+0000 7ff263684640 10 monclient: monmap: 2024-03-29T11:14:45.324 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:14:45.324 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:45.324 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:14:45.325 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:14:45.325 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:45.325 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:45.325 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:14:45.325 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:14:45.325 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:14:45.325 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:45.328 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.347+0000 7ff263684640 1 librados: starting msgr at 2024-03-29T11:14:45.328 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.347+0000 7ff263684640 1 librados: starting objecter 2024-03-29T11:14:45.328 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.347+0000 7ff263684640 1 Processor -- start 2024-03-29T11:14:45.328 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.347+0000 7ff263684640 1 -- start start 2024-03-29T11:14:45.328 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.347+0000 7ff263684640 1 librados: setting wanted keys 2024-03-29T11:14:45.328 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.347+0000 7ff263684640 1 librados: calling monclient init 2024-03-29T11:14:45.328 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.347+0000 7ff263684640 10 monclient: init 2024-03-29T11:14:45.328 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.348+0000 7ff263684640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:45.329 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.348+0000 7ff263684640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:14:45.329 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.348+0000 7ff263684640 10 monclient(hunting): picked mon.noname-a con 0x7ff25c0a9d70 addr v1:172.21.15.18:6789/0 2024-03-29T11:14:45.329 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.348+0000 7ff263684640 10 monclient(hunting): picked mon.noname-b con 0x7ff25c13f650 addr v1:172.21.15.70:6789/0 2024-03-29T11:14:45.329 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.348+0000 7ff263684640 10 monclient(hunting): picked mon.noname-c con 0x7ff25c1457f0 addr v1:172.21.15.18:6790/0 2024-03-29T11:14:45.329 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.348+0000 7ff263684640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff25c1420a0 con 0x7ff25c0a9d70 2024-03-29T11:14:45.329 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.348+0000 7ff263684640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff25c0b13c0 con 0x7ff25c13f650 2024-03-29T11:14:45.329 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.348+0000 7ff263684640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff25c144aa0 con 0x7ff25c1457f0 2024-03-29T11:14:45.329 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.348+0000 7ff263684640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:45.329 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.348+0000 7ff263684640 10 monclient(hunting): authenticate will time out at 1111.620054s 2024-03-29T11:14:45.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.351+0000 7ff261420640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7ff25c0a9d70 0x7ff25c1427d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:32870/0 (socket says 172.21.15.18:32870) 2024-03-29T11:14:45.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.351+0000 7ff261420640 1 -- 172.21.15.18:0/3863340186 learned_addr learned my addr 172.21.15.18:0/3863340186 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:14:45.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.352+0000 7ff251ffb640 10 client.?.objecter ms_handle_connect 0x7ff25c13f650 2024-03-29T11:14:45.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.352+0000 7ff251ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:45.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.352+0000 7ff251ffb640 10 client.?.objecter ms_handle_connect 0x7ff25c0a9d70 2024-03-29T11:14:45.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.352+0000 7ff251ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:45.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.352+0000 7ff251ffb640 1 -- 172.21.15.18:0/3863340186 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3828884064 0 0) 0x7ff25c0b13c0 con 0x7ff25c13f650 2024-03-29T11:14:45.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.352+0000 7ff251ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:14:45.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.352+0000 7ff251ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:45.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.352+0000 7ff251ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:45.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.352+0000 7ff251ffb640 10 monclient(hunting): my global_id is 4396 2024-03-29T11:14:45.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.353+0000 7ff251ffb640 1 -- 172.21.15.18:0/3863340186 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff240004eb0 con 0x7ff25c13f650 2024-03-29T11:14:45.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.353+0000 7ff251ffb640 1 -- 172.21.15.18:0/3863340186 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1624514358 0 0) 0x7ff25c1420a0 con 0x7ff25c0a9d70 2024-03-29T11:14:45.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.353+0000 7ff251ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:14:45.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.353+0000 7ff251ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:45.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.353+0000 7ff251ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:45.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.353+0000 7ff251ffb640 10 monclient(hunting): my global_id is 4413 2024-03-29T11:14:45.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.353+0000 7ff251ffb640 1 -- 172.21.15.18:0/3863340186 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff25c0b13c0 con 0x7ff25c0a9d70 2024-03-29T11:14:45.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.353+0000 7ff251ffb640 1 -- 172.21.15.18:0/3863340186 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2718941196 0 0) 0x7ff240004eb0 con 0x7ff25c13f650 2024-03-29T11:14:45.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.353+0000 7ff251ffb640 1 -- 172.21.15.18:0/3863340186 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff25c1420a0 con 0x7ff25c13f650 2024-03-29T11:14:45.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.353+0000 7ff251ffb640 1 -- 172.21.15.18:0/3863340186 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7ff258001630 con 0x7ff25c13f650 2024-03-29T11:14:45.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.353+0000 7ff251ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:45.334 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.353+0000 7ff251ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:45.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.353+0000 7ff251ffb640 10 monclient(hunting): dump: 2024-03-29T11:14:45.335 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:14:45.335 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:45.335 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:45.335 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:45.335 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:45.335 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:45.336 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:14:45.336 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:14:45.336 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:14:45.336 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:45.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.353+0000 7ff251ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:45.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.354+0000 7ff251ffb640 1 -- 172.21.15.18:0/3863340186 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2016133694 0 0) 0x7ff25c0b13c0 con 0x7ff25c0a9d70 2024-03-29T11:14:45.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.354+0000 7ff251ffb640 1 -- 172.21.15.18:0/3863340186 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff240004eb0 con 0x7ff25c0a9d70 2024-03-29T11:14:45.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.354+0000 7ff251ffb640 1 -- 172.21.15.18:0/3863340186 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7ff24c003e70 con 0x7ff25c0a9d70 2024-03-29T11:14:45.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.354+0000 7ff251ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:45.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.354+0000 7ff251ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:14:45.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.354+0000 7ff251ffb640 10 monclient(hunting): dump: 2024-03-29T11:14:45.336 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:14:45.336 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:45.336 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:45.336 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:45.337 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:45.337 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:45.337 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:14:45.337 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:14:45.337 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:14:45.337 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:45.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.355+0000 7ff251ffb640 1 -- 172.21.15.18:0/3863340186 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2848010324 0 0) 0x7ff240004eb0 con 0x7ff25c0a9d70 2024-03-29T11:14:45.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.355+0000 7ff251ffb640 1 -- 172.21.15.18:0/3863340186 >> v1:172.21.15.18:6790/0 conn(0x7ff25c1457f0 legacy=0x7ff25c1e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:45.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.355+0000 7ff251ffb640 1 -- 172.21.15.18:0/3863340186 >> v1:172.21.15.70:6789/0 conn(0x7ff25c13f650 legacy=0x7ff25c141990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:45.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.355+0000 7ff251ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:14:45.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.355+0000 7ff251ffb640 1 monclient: found mon.a 2024-03-29T11:14:45.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.355+0000 7ff251ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:45.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.355+0000 7ff251ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:14:45.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.355+0000 7ff251ffb640 1 -- 172.21.15.18:0/3863340186 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff25c144c30 con 0x7ff25c0a9d70 2024-03-29T11:14:45.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.355+0000 7ff251ffb640 10 monclient: _finish_auth 0 2024-03-29T11:14:45.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.355+0000 7ff251ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:14:45.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.355+0000 7ff251ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:45.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.356+0000 7ff251ffb640 1 -- 172.21.15.18:0/3863340186 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff24c004380 con 0x7ff25c0a9d70 2024-03-29T11:14:45.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.356+0000 7ff251ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:45.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.356+0000 7ff251ffb640 1 -- 172.21.15.18:0/3863340186 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7ff24c001070 con 0x7ff25c0a9d70 2024-03-29T11:14:45.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.356+0000 7ff2537fe640 4 set_mon_vals no callback set 2024-03-29T11:14:45.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.356+0000 7ff251ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:45.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.356+0000 7ff251ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:14:45.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.356+0000 7ff251ffb640 10 monclient: dump: 2024-03-29T11:14:45.342 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:14:45.342 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:45.342 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:45.342 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:45.342 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:45.342 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:45.342 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:14:45.342 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:14:45.342 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:14:45.342 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:45.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.356+0000 7ff263684640 5 monclient: authenticate success, global_id 4413 2024-03-29T11:14:45.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.356+0000 7ff263684640 10 monclient: _renew_subs 2024-03-29T11:14:45.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.356+0000 7ff263684640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:14:45.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.356+0000 7ff263684640 1 -- 172.21.15.18:0/3863340186 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff25c1d7570 con 0x7ff25c0a9d70 2024-03-29T11:14:45.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.357+0000 7ff263684640 10 client.4413.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:45.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.357+0000 7ff263684640 10 monclient: _renew_subs 2024-03-29T11:14:45.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.357+0000 7ff263684640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:14:45.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.357+0000 7ff263684640 1 -- 172.21.15.18:0/3863340186 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff25c1e3760 con 0x7ff25c0a9d70 2024-03-29T11:14:45.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.357+0000 7ff263684640 1 librados: init done 2024-03-29T11:14:45.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.359+0000 7ff263684640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:45.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.359+0000 7ff263684640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:45.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.359+0000 7ff263684640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:14:45.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.359+0000 7ff263684640 1 -- 172.21.15.18:0/3863340186 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff25c142410 con 0x7ff25c0a9d70 2024-03-29T11:14:45.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.359+0000 7ff251ffb640 1 -- 172.21.15.18:0/3863340186 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7ff24c0011d0 con 0x7ff25c0a9d70 2024-03-29T11:14:45.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.360+0000 7ff251ffb640 1 -- 172.21.15.18:0/3863340186 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7ff24c084870 con 0x7ff25c0a9d70 2024-03-29T11:14:45.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.360+0000 7ff251ffb640 10 client.4413.objecter ms_dispatch 0x7ff25c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:45.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.360+0000 7ff251ffb640 3 client.4413.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:45.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.360+0000 7ff251ffb640 3 client.4413.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:45.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.360+0000 7ff251ffb640 20 client.4413.objecter dump_active .. 0 homeless 2024-03-29T11:14:45.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.360+0000 7ff260c1f640 10 monclient: get_auth_request con 0x7ff24006dff0 auth_method 0 2024-03-29T11:14:45.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.361+0000 7ff251ffb640 10 client.4413.objecter ms_handle_connect 0x7ff24006dff0 2024-03-29T11:14:45.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.368+0000 7ff251ffb640 1 -- 172.21.15.18:0/3863340186 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff24c050150 con 0x7ff25c0a9d70 2024-03-29T11:14:45.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.368+0000 7ff251ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:45.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.368+0000 7ff251ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:45.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.695+0000 7ff263684640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:45.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.695+0000 7ff263684640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:45.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.695+0000 7ff263684640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:14:45.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.695+0000 7ff263684640 1 -- 172.21.15.18:0/3863340186 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff25c1d7700 con 0x7ff25c0a9d70 2024-03-29T11:14:45.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.697+0000 7ff251ffb640 1 -- 172.21.15.18:0/3863340186 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7ff24c052080 con 0x7ff25c0a9d70 2024-03-29T11:14:45.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.697+0000 7ff251ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:45.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.697+0000 7ff251ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:45.675 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:14:45.675 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:45.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.701+0000 7ff263684640 10 librados: watch_flush enter 2024-03-29T11:14:45.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.701+0000 7ff263684640 10 librados: watch_flush exit 2024-03-29T11:14:45.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.701+0000 7ff263684640 20 client.4413.objecter shutdown clearing up homeless session... 2024-03-29T11:14:45.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.701+0000 7ff263684640 10 client.4413.objecter successfully canceled tick 2024-03-29T11:14:45.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.701+0000 7ff263684640 1 -- 172.21.15.18:0/3863340186 >> v1:172.21.15.70:6816/3101871562 conn(0x7ff24006dff0 legacy=0x7ff2400704b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:45.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.701+0000 7ff263684640 10 monclient: shutdown 2024-03-29T11:14:45.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.701+0000 7ff263684640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:45.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.701+0000 7ff263684640 1 -- 172.21.15.18:0/3863340186 >> v1:172.21.15.18:6789/0 conn(0x7ff25c0a9d70 legacy=0x7ff25c1427d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:45.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.702+0000 7ff263684640 1 -- 172.21.15.18:0/3863340186 shutdown_connections 2024-03-29T11:14:45.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.702+0000 7ff263684640 1 -- 172.21.15.18:0/3863340186 >> 172.21.15.18:0/3863340186 conn(0x7ff25c066a00 msgr2=0x7ff25c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:45.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.703+0000 7ff263684640 1 -- 172.21.15.18:0/3863340186 shutdown_connections 2024-03-29T11:14:45.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.703+0000 7ff263684640 1 -- 172.21.15.18:0/3863340186 wait complete. 2024-03-29T11:14:45.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:45.703+0000 7ff263684640 1 librados: shutdown 2024-03-29T11:14:45.698 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:52.506 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:14:52.652 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.674+0000 7f38f4832640 10 monclient: get_monmap_and_config 2024-03-29T11:14:52.652 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.674+0000 7f38f4832640 10 monclient: build_initial_monmap 2024-03-29T11:14:52.652 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.674+0000 7f38f4832640 10 monclient: monmap: 2024-03-29T11:14:52.652 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:14:52.652 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:52.652 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:14:52.675484+0000 2024-03-29T11:14:52.652 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:14:52.675484+0000 2024-03-29T11:14:52.652 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:52.652 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:52.652 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:14:52.653 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:14:52.653 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:14:52.653 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:52.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.675+0000 7f38f4832640 1 Processor -- start 2024-03-29T11:14:52.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.675+0000 7f38f4832640 1 -- start start 2024-03-29T11:14:52.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.675+0000 7f38f4832640 10 monclient: init 2024-03-29T11:14:52.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.677+0000 7f38f4832640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:52.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.677+0000 7f38f4832640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:14:52.656 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.678+0000 7f38f4832640 10 monclient(hunting): picked mon.noname-c con 0x7f38f00b2800 addr v1:172.21.15.70:6789/0 2024-03-29T11:14:52.656 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.678+0000 7f38f4832640 10 monclient(hunting): picked mon.noname-a con 0x7f38f00a9c00 addr v1:172.21.15.18:6789/0 2024-03-29T11:14:52.656 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.679+0000 7f38f4832640 10 monclient(hunting): picked mon.noname-b con 0x7f38f00b38c0 addr v1:172.21.15.18:6790/0 2024-03-29T11:14:52.656 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.679+0000 7f38f4832640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f0141eb0 con 0x7f38f00a9c00 2024-03-29T11:14:52.657 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.679+0000 7f38f4832640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f00b22d0 con 0x7f38f00b2800 2024-03-29T11:14:52.657 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.679+0000 7f38f4832640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f0142d30 con 0x7f38f00b38c0 2024-03-29T11:14:52.657 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.679+0000 7f38f4832640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:52.657 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.679+0000 7f38f4832640 10 monclient(hunting): authenticate will time out at 1118.950975s 2024-03-29T11:14:52.658 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.680+0000 7f38ef59e640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f38f00b38c0 0x7f38f014d7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:56405/0 (socket says 172.21.15.18:56405) 2024-03-29T11:14:52.658 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.680+0000 7f38ef59e640 1 -- 172.21.15.18:0/424673320 learned_addr learned my addr 172.21.15.18:0/424673320 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:14:52.658 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.681+0000 7f38ee59c640 1 -- 172.21.15.18:0/424673320 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 70382008 0 0) 0x7f38f0142d30 con 0x7f38f00b38c0 2024-03-29T11:14:52.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.681+0000 7f38ee59c640 10 monclient(hunting): _negotiate 2024-03-29T11:14:52.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.681+0000 7f38ee59c640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:52.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.681+0000 7f38ee59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:52.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.681+0000 7f38ee59c640 10 monclient(hunting): my global_id is 4466 2024-03-29T11:14:52.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.681+0000 7f38ee59c640 1 -- 172.21.15.18:0/424673320 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38dc003520 con 0x7f38f00b38c0 2024-03-29T11:14:52.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.681+0000 7f38ee59c640 1 -- 172.21.15.18:0/424673320 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 470427820 0 0) 0x7f38dc003520 con 0x7f38f00b38c0 2024-03-29T11:14:52.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.681+0000 7f38ee59c640 1 -- 172.21.15.18:0/424673320 >> v1:172.21.15.70:6789/0 conn(0x7f38f00b2800 legacy=0x7f38f0142620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:14:52.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.684+0000 7f38ee59c640 1 -- 172.21.15.18:0/424673320 >> v1:172.21.15.18:6789/0 conn(0x7f38f00a9c00 legacy=0x7f38f013bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:52.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.684+0000 7f38ee59c640 10 monclient: _finish_hunting 0 2024-03-29T11:14:52.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.685+0000 7f38ee59c640 1 monclient: found mon.noname-b 2024-03-29T11:14:52.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.685+0000 7f38ee59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:52.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.685+0000 7f38ee59c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:14:52.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.685+0000 7f38ee59c640 1 -- 172.21.15.18:0/424673320 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f00aaf90 con 0x7f38f00b38c0 2024-03-29T11:14:52.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.685+0000 7f38ee59c640 10 monclient: _finish_auth 0 2024-03-29T11:14:52.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.685+0000 7f38ee59c640 10 monclient: _check_auth_tickets 2024-03-29T11:14:52.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.685+0000 7f38ee59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:52.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.685+0000 7f38ee59c640 1 -- 172.21.15.18:0/424673320 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f38e8001a10 con 0x7f38f00b38c0 2024-03-29T11:14:52.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.685+0000 7f38ee59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:52.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.685+0000 7f38ee59c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:14:52.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.685+0000 7f38ee59c640 10 monclient: dump: 2024-03-29T11:14:52.665 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:14:52.665 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:52.665 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:52.665 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:52.665 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:52.665 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:52.665 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:14:52.665 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:14:52.665 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:14:52.665 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:52.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.685+0000 7f38ee59c640 1 -- 172.21.15.18:0/424673320 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38e8001c30 con 0x7f38f00b38c0 2024-03-29T11:14:52.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.685+0000 7f38ee59c640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:52.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.686+0000 7f38ee59c640 1 -- 172.21.15.18:0/424673320 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f38e8001fc0 con 0x7f38f00b38c0 2024-03-29T11:14:52.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.686+0000 7f38ee59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:52.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.686+0000 7f38ee59c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:14:52.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.686+0000 7f38ee59c640 10 monclient: dump: 2024-03-29T11:14:52.666 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:14:52.666 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:52.667 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:52.667 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:52.667 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:52.667 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:52.667 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:14:52.667 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:14:52.667 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:14:52.667 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:52.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.686+0000 7f38f4832640 5 monclient: authenticate success, global_id 4466 2024-03-29T11:14:52.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.686+0000 7f38f4832640 10 monclient: get_monmap_and_config success 2024-03-29T11:14:52.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.686+0000 7f38f4832640 4 set_mon_vals no callback set 2024-03-29T11:14:52.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.687+0000 7f38f4832640 10 monclient: shutdown 2024-03-29T11:14:52.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.687+0000 7f38f4832640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:52.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.687+0000 7f38f4832640 1 -- 172.21.15.18:0/424673320 >> v1:172.21.15.18:6790/0 conn(0x7f38f00b38c0 legacy=0x7f38f014d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:52.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.687+0000 7f38f4832640 1 -- 172.21.15.18:0/424673320 shutdown_connections 2024-03-29T11:14:52.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.687+0000 7f38f4832640 1 -- 172.21.15.18:0/424673320 >> 172.21.15.18:0/424673320 conn(0x7f38f0066a00 msgr2=0x7f38f0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:52.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.689+0000 7f38f4832640 1 -- 172.21.15.18:0/424673320 shutdown_connections 2024-03-29T11:14:52.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.689+0000 7f38f4832640 1 -- 172.21.15.18:0/424673320 wait complete. 2024-03-29T11:14:52.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.690+0000 7f38f4832640 10 monclient: build_initial_monmap 2024-03-29T11:14:52.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.690+0000 7f38f4832640 10 monclient: monmap: 2024-03-29T11:14:52.670 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:14:52.670 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:14:52.670 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:14:52.670 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:14:52.670 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:14:52.670 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:52.670 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:14:52.670 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:14:52.670 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:14:52.670 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:52.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.691+0000 7f38f4832640 1 librados: starting msgr at 2024-03-29T11:14:52.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.691+0000 7f38f4832640 1 librados: starting objecter 2024-03-29T11:14:52.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.691+0000 7f38f4832640 1 Processor -- start 2024-03-29T11:14:52.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.691+0000 7f38f4832640 1 -- start start 2024-03-29T11:14:52.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.691+0000 7f38f4832640 1 librados: setting wanted keys 2024-03-29T11:14:52.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.692+0000 7f38f4832640 1 librados: calling monclient init 2024-03-29T11:14:52.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.692+0000 7f38f4832640 10 monclient: init 2024-03-29T11:14:52.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.692+0000 7f38f4832640 10 monclient: _reopen_session rank -1 2024-03-29T11:14:52.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.693+0000 7f38f4832640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:14:52.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.693+0000 7f38f4832640 10 monclient(hunting): picked mon.noname-c con 0x7f38f00a9c00 addr v1:172.21.15.18:6790/0 2024-03-29T11:14:52.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.695+0000 7f38f4832640 10 monclient(hunting): picked mon.noname-b con 0x7f38f00b2800 addr v1:172.21.15.70:6789/0 2024-03-29T11:14:52.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.695+0000 7f38f4832640 10 monclient(hunting): picked mon.noname-a con 0x7f38f00b38c0 addr v1:172.21.15.18:6789/0 2024-03-29T11:14:52.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.695+0000 7f38f4832640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f013f210 con 0x7f38f00b38c0 2024-03-29T11:14:52.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.695+0000 7f38f4832640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f00ac4f0 con 0x7f38f00b2800 2024-03-29T11:14:52.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.695+0000 7f38f4832640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38f00ac7d0 con 0x7f38f00a9c00 2024-03-29T11:14:52.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.695+0000 7f38f4832640 10 monclient(hunting): _renew_subs 2024-03-29T11:14:52.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.695+0000 7f38f4832640 10 monclient(hunting): authenticate will time out at 1118.967308s 2024-03-29T11:14:52.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.695+0000 7f38eed9d640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f38f00a9c00 0x7f38f013eb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:56513/0 (socket says 172.21.15.18:56513) 2024-03-29T11:14:52.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.695+0000 7f38eed9d640 1 -- 172.21.15.18:0/4222568922 learned_addr learned my addr 172.21.15.18:0/4222568922 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:14:52.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.695+0000 7f38e77fe640 10 client.?.objecter ms_handle_connect 0x7f38f00a9c00 2024-03-29T11:14:52.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.695+0000 7f38e77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:52.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.696+0000 7f38e77fe640 1 -- 172.21.15.18:0/4222568922 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1821858467 0 0) 0x7f38f00ac7d0 con 0x7f38f00a9c00 2024-03-29T11:14:52.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.696+0000 7f38e77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:14:52.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.696+0000 7f38e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:52.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.696+0000 7f38e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:52.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.696+0000 7f38e77fe640 10 monclient(hunting): my global_id is 4469 2024-03-29T11:14:52.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.696+0000 7f38e77fe640 1 -- 172.21.15.18:0/4222568922 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38d0004eb0 con 0x7f38f00a9c00 2024-03-29T11:14:52.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.696+0000 7f38e77fe640 10 client.?.objecter ms_handle_connect 0x7f38f00b38c0 2024-03-29T11:14:52.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.696+0000 7f38e77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:14:52.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.696+0000 7f38e77fe640 1 -- 172.21.15.18:0/4222568922 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1153316576 0 0) 0x7f38d0004eb0 con 0x7f38f00a9c00 2024-03-29T11:14:52.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.696+0000 7f38e77fe640 1 -- 172.21.15.18:0/4222568922 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38f00ac7d0 con 0x7f38f00a9c00 2024-03-29T11:14:52.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.697+0000 7f38e77fe640 1 -- 172.21.15.18:0/4222568922 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2095478214 0 0) 0x7f38f013f210 con 0x7f38f00b38c0 2024-03-29T11:14:52.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.697+0000 7f38e77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:14:52.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.697+0000 7f38e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:14:52.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.697+0000 7f38e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:14:52.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.697+0000 7f38e77fe640 10 monclient(hunting): my global_id is 4419 2024-03-29T11:14:52.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.697+0000 7f38e77fe640 1 -- 172.21.15.18:0/4222568922 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38d0005590 con 0x7f38f00b38c0 2024-03-29T11:14:52.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.697+0000 7f38e77fe640 1 -- 172.21.15.18:0/4222568922 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2454637821 0 0) 0x7f38d0005590 con 0x7f38f00b38c0 2024-03-29T11:14:52.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.697+0000 7f38e77fe640 1 -- 172.21.15.18:0/4222568922 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38f013f210 con 0x7f38f00b38c0 2024-03-29T11:14:52.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.697+0000 7f38e77fe640 1 -- 172.21.15.18:0/4222568922 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f38e80018d0 con 0x7f38f00b38c0 2024-03-29T11:14:52.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.697+0000 7f38e77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:52.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.697+0000 7f38e77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:14:52.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.697+0000 7f38e77fe640 10 monclient(hunting): dump: 2024-03-29T11:14:52.677 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:14:52.677 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:52.677 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:52.677 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:52.678 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:52.678 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:52.678 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:14:52.678 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:14:52.678 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:14:52.678 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:52.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.697+0000 7f38e77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:14:52.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.697+0000 7f38e77fe640 1 -- 172.21.15.18:0/4222568922 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1640035442 0 0) 0x7f38f013f210 con 0x7f38f00b38c0 2024-03-29T11:14:52.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.697+0000 7f38e77fe640 1 -- 172.21.15.18:0/4222568922 >> v1:172.21.15.18:6790/0 conn(0x7f38f00a9c00 legacy=0x7f38f013eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:52.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.698+0000 7f38e77fe640 1 -- 172.21.15.18:0/4222568922 >> v1:172.21.15.70:6789/0 conn(0x7f38f00b2800 legacy=0x7f38f0137cb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:14:52.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.698+0000 7f38e77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:14:52.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.698+0000 7f38e77fe640 1 monclient: found mon.a 2024-03-29T11:14:52.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.698+0000 7f38e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:14:52.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.698+0000 7f38e77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:14:52.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.698+0000 7f38e77fe640 1 -- 172.21.15.18:0/4222568922 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38f013f440 con 0x7f38f00b38c0 2024-03-29T11:14:52.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.698+0000 7f38e77fe640 10 monclient: _finish_auth 0 2024-03-29T11:14:52.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.698+0000 7f38e77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:14:52.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.698+0000 7f38e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:14:52.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.698+0000 7f38f4832640 5 monclient: authenticate success, global_id 4419 2024-03-29T11:14:52.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.698+0000 7f38f4832640 10 monclient: _renew_subs 2024-03-29T11:14:52.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.698+0000 7f38f4832640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:14:52.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.698+0000 7f38f4832640 1 -- 172.21.15.18:0/4222568922 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38f00ab370 con 0x7f38f00b38c0 2024-03-29T11:14:52.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.699+0000 7f38e77fe640 1 -- 172.21.15.18:0/4222568922 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38e8001a30 con 0x7f38f00b38c0 2024-03-29T11:14:52.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.699+0000 7f38e77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:14:52.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.699+0000 7f38e77fe640 1 -- 172.21.15.18:0/4222568922 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f38e8004fb0 con 0x7f38f00b38c0 2024-03-29T11:14:52.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.699+0000 7f38e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:14:52.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.699+0000 7f38e77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:14:52.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.699+0000 7f38e77fe640 10 monclient: dump: 2024-03-29T11:14:52.681 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:14:52.682 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:14:52.682 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:52.682 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:14:52.682 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:14:52.682 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:14:52.682 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:14:52.682 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:14:52.682 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:14:52.682 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:14:52.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.699+0000 7f38f4832640 10 client.4419.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:14:52.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.699+0000 7f38f4832640 10 monclient: _renew_subs 2024-03-29T11:14:52.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.699+0000 7f38f4832640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:14:52.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.699+0000 7f38f4832640 1 -- 172.21.15.18:0/4222568922 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f38f01385a0 con 0x7f38f00b38c0 2024-03-29T11:14:52.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.699+0000 7f38f4832640 1 librados: init done 2024-03-29T11:14:52.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.699+0000 7f38ed59a640 4 set_mon_vals no callback set 2024-03-29T11:14:52.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.701+0000 7f38e77fe640 1 -- 172.21.15.18:0/4222568922 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f38e801af80 con 0x7f38f00b38c0 2024-03-29T11:14:52.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.702+0000 7f38e77fe640 1 -- 172.21.15.18:0/4222568922 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f38e8084d90 con 0x7f38f00b38c0 2024-03-29T11:14:52.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.702+0000 7f38e77fe640 10 client.4419.objecter ms_dispatch 0x7f38f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:14:52.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.702+0000 7f38e77fe640 3 client.4419.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:14:52.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.702+0000 7f38e77fe640 3 client.4419.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:14:52.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.702+0000 7f38e77fe640 20 client.4419.objecter dump_active .. 0 homeless 2024-03-29T11:14:52.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.702+0000 7f38f4832640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:52.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.702+0000 7f38f4832640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:52.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.702+0000 7f38f4832640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:14:52.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.702+0000 7f38f4832640 1 -- 172.21.15.18:0/4222568922 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38d4005080 con 0x7f38f00b38c0 2024-03-29T11:14:52.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.702+0000 7f38eed9d640 10 monclient: get_auth_request con 0x7f38d006ddd0 auth_method 0 2024-03-29T11:14:52.684 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.705+0000 7f38e77fe640 10 client.4419.objecter ms_handle_connect 0x7f38d006ddd0 2024-03-29T11:14:52.689 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.711+0000 7f38e77fe640 1 -- 172.21.15.18:0/4222568922 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f38e8050670 con 0x7f38f00b38c0 2024-03-29T11:14:52.690 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.711+0000 7f38e77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:14:52.690 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:52.711+0000 7f38e77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:14:53.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:53.054+0000 7f38f4832640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:53.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:53.054+0000 7f38f4832640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:53.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:53.054+0000 7f38f4832640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:14:53.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:53.054+0000 7f38f4832640 1 -- 172.21.15.18:0/4222568922 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f38d4005330 con 0x7f38f00b38c0 2024-03-29T11:14:53.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:53.056+0000 7f38e77fe640 1 -- 172.21.15.18:0/4222568922 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f38e80525a0 con 0x7f38f00b38c0 2024-03-29T11:14:53.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:53.056+0000 7f38e77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:14:53.034 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:53.056+0000 7f38e77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:14:53.035 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:14:53.036 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:14:53.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:53.059+0000 7f38f4832640 10 librados: watch_flush enter 2024-03-29T11:14:53.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:53.059+0000 7f38f4832640 10 librados: watch_flush exit 2024-03-29T11:14:53.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:53.060+0000 7f38f4832640 20 client.4419.objecter shutdown clearing up homeless session... 2024-03-29T11:14:53.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:53.060+0000 7f38f4832640 10 client.4419.objecter successfully canceled tick 2024-03-29T11:14:53.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:53.060+0000 7f38f4832640 1 -- 172.21.15.18:0/4222568922 >> v1:172.21.15.70:6816/3101871562 conn(0x7f38d006ddd0 legacy=0x7f38d0070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:53.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:53.060+0000 7f38f4832640 10 monclient: shutdown 2024-03-29T11:14:53.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:53.060+0000 7f38f4832640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:14:53.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:53.060+0000 7f38f4832640 1 -- 172.21.15.18:0/4222568922 >> v1:172.21.15.18:6789/0 conn(0x7f38f00b38c0 legacy=0x7f38f01da240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:14:53.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:53.060+0000 7f38f4832640 1 -- 172.21.15.18:0/4222568922 shutdown_connections 2024-03-29T11:14:53.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:53.060+0000 7f38f4832640 1 -- 172.21.15.18:0/4222568922 >> 172.21.15.18:0/4222568922 conn(0x7f38f0066a00 msgr2=0x7f38f00b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:14:53.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:53.060+0000 7f38f4832640 1 -- 172.21.15.18:0/4222568922 shutdown_connections 2024-03-29T11:14:53.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:53.060+0000 7f38f4832640 1 -- 172.21.15.18:0/4222568922 wait complete. 2024-03-29T11:14:53.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:14:53.060+0000 7f38f4832640 1 librados: shutdown 2024-03-29T11:14:53.055 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:14:59.863 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:00.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.027+0000 7f29ade33640 10 monclient: get_monmap_and_config 2024-03-29T11:15:00.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.027+0000 7f29ade33640 10 monclient: build_initial_monmap 2024-03-29T11:15:00.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.027+0000 7f29ade33640 10 monclient: monmap: 2024-03-29T11:15:00.005 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:15:00.005 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:00.005 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:15:00.028423+0000 2024-03-29T11:15:00.005 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:15:00.028423+0000 2024-03-29T11:15:00.005 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:00.005 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:00.005 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:15:00.006 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:15:00.006 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:15:00.006 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:00.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.028+0000 7f29ade33640 1 Processor -- start 2024-03-29T11:15:00.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.028+0000 7f29ade33640 1 -- start start 2024-03-29T11:15:00.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.028+0000 7f29ade33640 10 monclient: init 2024-03-29T11:15:00.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.029+0000 7f29ade33640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:00.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.029+0000 7f29ade33640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:15:00.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.029+0000 7f29ade33640 10 monclient(hunting): picked mon.noname-b con 0x7f29a81422b0 addr v1:172.21.15.18:6790/0 2024-03-29T11:15:00.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.029+0000 7f29ade33640 10 monclient(hunting): picked mon.noname-c con 0x7f29a8142690 addr v1:172.21.15.70:6789/0 2024-03-29T11:15:00.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.029+0000 7f29ade33640 10 monclient(hunting): picked mon.noname-a con 0x7f29a80b17d0 addr v1:172.21.15.18:6789/0 2024-03-29T11:15:00.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.029+0000 7f29ade33640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a8142c80 con 0x7f29a80b17d0 2024-03-29T11:15:00.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.029+0000 7f29ade33640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a80aae40 con 0x7f29a8142690 2024-03-29T11:15:00.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.029+0000 7f29ade33640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a813fbd0 con 0x7f29a81422b0 2024-03-29T11:15:00.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.029+0000 7f29ade33640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:00.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.029+0000 7f29ade33640 10 monclient(hunting): authenticate will time out at 1126.301548s 2024-03-29T11:15:00.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.029+0000 7f29a77fe640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f29a81422b0 0x7f29a813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:55210/0 (socket says 172.21.15.18:55210) 2024-03-29T11:15:00.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.029+0000 7f29a77fe640 1 -- 172.21.15.18:0/1536302796 learned_addr learned my addr 172.21.15.18:0/1536302796 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:15:00.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.030+0000 7f29a67fc640 1 -- 172.21.15.18:0/1536302796 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3704705262 0 0) 0x7f29a813fbd0 con 0x7f29a81422b0 2024-03-29T11:15:00.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.030+0000 7f29a67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:15:00.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.030+0000 7f29a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:00.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.030+0000 7f29a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:00.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.030+0000 7f29a67fc640 10 monclient(hunting): my global_id is 4472 2024-03-29T11:15:00.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.030+0000 7f29a67fc640 1 -- 172.21.15.18:0/1536302796 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2990003520 con 0x7f29a81422b0 2024-03-29T11:15:00.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.030+0000 7f29a67fc640 1 -- 172.21.15.18:0/1536302796 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3630874926 0 0) 0x7f2990003520 con 0x7f29a81422b0 2024-03-29T11:15:00.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.030+0000 7f29a67fc640 1 -- 172.21.15.18:0/1536302796 >> v1:172.21.15.70:6789/0 conn(0x7f29a8142690 legacy=0x7f29a80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:15:00.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.030+0000 7f29a67fc640 1 -- 172.21.15.18:0/1536302796 >> v1:172.21.15.18:6789/0 conn(0x7f29a80b17d0 legacy=0x7f29a80b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:15:00.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.032+0000 7f29a67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:15:00.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.032+0000 7f29a67fc640 1 monclient: found mon.noname-b 2024-03-29T11:15:00.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.032+0000 7f29a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:00.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.032+0000 7f29a67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:15:00.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.032+0000 7f29a67fc640 1 -- 172.21.15.18:0/1536302796 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a8151940 con 0x7f29a81422b0 2024-03-29T11:15:00.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.032+0000 7f29a67fc640 10 monclient: _finish_auth 0 2024-03-29T11:15:00.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.032+0000 7f29a67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:15:00.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.032+0000 7f29a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:00.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.032+0000 7f29ade33640 5 monclient: authenticate success, global_id 4472 2024-03-29T11:15:00.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.032+0000 7f29ade33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:00.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.032+0000 7f29a67fc640 1 -- 172.21.15.18:0/1536302796 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f299c001690 con 0x7f29a81422b0 2024-03-29T11:15:00.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.032+0000 7f29a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:00.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.032+0000 7f29a67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:15:00.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.032+0000 7f29a67fc640 10 monclient: dump: 2024-03-29T11:15:00.012 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:15:00.013 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:00.013 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:00.013 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:00.013 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:00.013 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:00.013 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:15:00.013 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:15:00.013 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:15:00.013 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:00.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.032+0000 7f29ade33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:00.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.032+0000 7f29a67fc640 1 -- 172.21.15.18:0/1536302796 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f299c0018b0 con 0x7f29a81422b0 2024-03-29T11:15:00.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.032+0000 7f29a67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:00.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.032+0000 7f29a67fc640 1 -- 172.21.15.18:0/1536302796 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f299c001c40 con 0x7f29a81422b0 2024-03-29T11:15:00.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.032+0000 7f29a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:00.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.032+0000 7f29a67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:00.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.032+0000 7f29a67fc640 10 monclient: dump: 2024-03-29T11:15:00.014 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:15:00.014 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:00.014 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:00.014 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:00.014 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:00.014 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:00.014 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:15:00.014 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:15:00.014 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:15:00.015 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:00.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.032+0000 7f29ade33640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:00.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.032+0000 7f29ade33640 4 set_mon_vals no callback set 2024-03-29T11:15:00.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.033+0000 7f29ade33640 10 monclient: shutdown 2024-03-29T11:15:00.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.033+0000 7f29ade33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:00.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.033+0000 7f29ade33640 1 -- 172.21.15.18:0/1536302796 >> v1:172.21.15.18:6790/0 conn(0x7f29a81422b0 legacy=0x7f29a813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:00.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.033+0000 7f29ade33640 1 -- 172.21.15.18:0/1536302796 shutdown_connections 2024-03-29T11:15:00.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.033+0000 7f29ade33640 1 -- 172.21.15.18:0/1536302796 >> 172.21.15.18:0/1536302796 conn(0x7f29a8066a00 msgr2=0x7f29a8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:00.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.033+0000 7f29ade33640 1 -- 172.21.15.18:0/1536302796 shutdown_connections 2024-03-29T11:15:00.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.033+0000 7f29ade33640 1 -- 172.21.15.18:0/1536302796 wait complete. 2024-03-29T11:15:00.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.034+0000 7f29ade33640 10 monclient: build_initial_monmap 2024-03-29T11:15:00.016 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.034+0000 7f29ade33640 10 monclient: monmap: 2024-03-29T11:15:00.016 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:15:00.016 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:00.016 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:15:00.016 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:15:00.016 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:00.016 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:00.016 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:15:00.016 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:15:00.016 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:15:00.016 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:00.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.035+0000 7f29ade33640 1 librados: starting msgr at 2024-03-29T11:15:00.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.035+0000 7f29ade33640 1 librados: starting objecter 2024-03-29T11:15:00.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.035+0000 7f29ade33640 1 Processor -- start 2024-03-29T11:15:00.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.035+0000 7f29ade33640 1 -- start start 2024-03-29T11:15:00.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.035+0000 7f29ade33640 1 librados: setting wanted keys 2024-03-29T11:15:00.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.035+0000 7f29ade33640 1 librados: calling monclient init 2024-03-29T11:15:00.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.035+0000 7f29ade33640 10 monclient: init 2024-03-29T11:15:00.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.036+0000 7f29ade33640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:00.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.036+0000 7f29ade33640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:15:00.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.036+0000 7f29ade33640 10 monclient(hunting): picked mon.noname-c con 0x7f29a80b17d0 addr v1:172.21.15.18:6790/0 2024-03-29T11:15:00.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.036+0000 7f29ade33640 10 monclient(hunting): picked mon.noname-a con 0x7f29a81422b0 addr v1:172.21.15.18:6789/0 2024-03-29T11:15:00.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.036+0000 7f29ade33640 10 monclient(hunting): picked mon.noname-b con 0x7f29a8142690 addr v1:172.21.15.70:6789/0 2024-03-29T11:15:00.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.036+0000 7f29ade33640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a8131e20 con 0x7f29a81422b0 2024-03-29T11:15:00.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.036+0000 7f29ade33640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a8132a30 con 0x7f29a8142690 2024-03-29T11:15:00.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.037+0000 7f29ade33640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a812f680 con 0x7f29a80b17d0 2024-03-29T11:15:00.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.037+0000 7f29ade33640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:00.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.037+0000 7f29ade33640 10 monclient(hunting): authenticate will time out at 1126.309025s 2024-03-29T11:15:00.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.037+0000 7f29a77fe640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f29a80b17d0 0x7f29a8132320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:34390/0 (socket says 172.21.15.18:34390) 2024-03-29T11:15:00.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.037+0000 7f29a77fe640 1 -- 172.21.15.18:0/2699189283 learned_addr learned my addr 172.21.15.18:0/2699189283 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:15:00.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.037+0000 7f298ffff640 10 client.?.objecter ms_handle_connect 0x7f29a80b17d0 2024-03-29T11:15:00.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.037+0000 7f298ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:00.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.037+0000 7f298ffff640 10 client.?.objecter ms_handle_connect 0x7f29a8142690 2024-03-29T11:15:00.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.037+0000 7f298ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:00.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.037+0000 7f298ffff640 1 -- 172.21.15.18:0/2699189283 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2245764143 0 0) 0x7f29a812f680 con 0x7f29a80b17d0 2024-03-29T11:15:00.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.037+0000 7f298ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:15:00.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.037+0000 7f298ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:00.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.037+0000 7f298ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:00.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.037+0000 7f298ffff640 10 monclient(hunting): my global_id is 4475 2024-03-29T11:15:00.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.038+0000 7f298ffff640 1 -- 172.21.15.18:0/2699189283 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2994004eb0 con 0x7f29a80b17d0 2024-03-29T11:15:00.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.038+0000 7f298ffff640 1 -- 172.21.15.18:0/2699189283 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3268059504 0 0) 0x7f29a8132a30 con 0x7f29a8142690 2024-03-29T11:15:00.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.038+0000 7f298ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:15:00.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.038+0000 7f298ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:00.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.038+0000 7f298ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:00.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.038+0000 7f298ffff640 10 monclient(hunting): my global_id is 4402 2024-03-29T11:15:00.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.038+0000 7f298ffff640 1 -- 172.21.15.18:0/2699189283 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29a812f680 con 0x7f29a8142690 2024-03-29T11:15:00.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.038+0000 7f298ffff640 1 -- 172.21.15.18:0/2699189283 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3218532596 0 0) 0x7f2994004eb0 con 0x7f29a80b17d0 2024-03-29T11:15:00.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.038+0000 7f298ffff640 1 -- 172.21.15.18:0/2699189283 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29a8132a30 con 0x7f29a80b17d0 2024-03-29T11:15:00.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.038+0000 7f298ffff640 1 -- 172.21.15.18:0/2699189283 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f299c001550 con 0x7f29a80b17d0 2024-03-29T11:15:00.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.038+0000 7f298ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:00.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.038+0000 7f298ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:15:00.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.038+0000 7f298ffff640 10 monclient(hunting): dump: 2024-03-29T11:15:00.026 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:15:00.026 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:00.026 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:00.026 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:00.026 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:00.026 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:00.026 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:15:00.026 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:15:00.026 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:15:00.026 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:00.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.038+0000 7f298ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:00.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.039+0000 7f298ffff640 1 -- 172.21.15.18:0/2699189283 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 188920222 0 0) 0x7f29a8132a30 con 0x7f29a80b17d0 2024-03-29T11:15:00.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.039+0000 7f298ffff640 1 -- 172.21.15.18:0/2699189283 >> v1:172.21.15.70:6789/0 conn(0x7f29a8142690 legacy=0x7f29a812ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:00.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.039+0000 7f298ffff640 1 -- 172.21.15.18:0/2699189283 >> v1:172.21.15.18:6789/0 conn(0x7f29a81422b0 legacy=0x7f29a812e860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:15:00.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.039+0000 7f298ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:15:00.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.039+0000 7f298ffff640 1 monclient: found mon.c 2024-03-29T11:15:00.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.039+0000 7f298ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:00.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.039+0000 7f298ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:15:00.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.039+0000 7f298ffff640 1 -- 172.21.15.18:0/2699189283 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a806eef0 con 0x7f29a80b17d0 2024-03-29T11:15:00.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.039+0000 7f298ffff640 10 monclient: _finish_auth 0 2024-03-29T11:15:00.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.039+0000 7f298ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:15:00.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.039+0000 7f298ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:00.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.039+0000 7f29ade33640 5 monclient: authenticate success, global_id 4475 2024-03-29T11:15:00.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.039+0000 7f29ade33640 10 monclient: _renew_subs 2024-03-29T11:15:00.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.039+0000 7f29ade33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:15:00.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.039+0000 7f29ade33640 1 -- 172.21.15.18:0/2699189283 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29a8131900 con 0x7f29a80b17d0 2024-03-29T11:15:00.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.039+0000 7f29ade33640 10 client.4475.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:00.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.040+0000 7f29ade33640 10 monclient: _renew_subs 2024-03-29T11:15:00.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.040+0000 7f29ade33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:15:00.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.040+0000 7f29ade33640 1 -- 172.21.15.18:0/2699189283 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f29a806e070 con 0x7f29a80b17d0 2024-03-29T11:15:00.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.040+0000 7f29ade33640 1 librados: init done 2024-03-29T11:15:00.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.040+0000 7f298ffff640 1 -- 172.21.15.18:0/2699189283 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f299c0016b0 con 0x7f29a80b17d0 2024-03-29T11:15:00.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.040+0000 7f298ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:00.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.040+0000 7f298ffff640 1 -- 172.21.15.18:0/2699189283 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f299c004d90 con 0x7f29a80b17d0 2024-03-29T11:15:00.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.040+0000 7f298ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:00.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.040+0000 7f298ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:00.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.040+0000 7f298ffff640 10 monclient: dump: 2024-03-29T11:15:00.028 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:15:00.028 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:00.028 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:00.028 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:00.029 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:00.029 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:00.029 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:15:00.029 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:15:00.029 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:15:00.029 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:00.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.040+0000 7f29a5ffb640 4 set_mon_vals no callback set 2024-03-29T11:15:00.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.042+0000 7f298ffff640 1 -- 172.21.15.18:0/2699189283 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f299c01ad60 con 0x7f29a80b17d0 2024-03-29T11:15:00.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.041+0000 7f29ade33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:00.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.042+0000 7f29ade33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:00.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.042+0000 7f29ade33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:15:00.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.042+0000 7f29ade33640 1 -- 172.21.15.18:0/2699189283 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2988005080 con 0x7f29a80b17d0 2024-03-29T11:15:00.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.043+0000 7f298ffff640 1 -- 172.21.15.18:0/2699189283 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f299c084c60 con 0x7f29a80b17d0 2024-03-29T11:15:00.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.043+0000 7f298ffff640 10 client.4475.objecter ms_dispatch 0x7f29a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:00.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.043+0000 7f298ffff640 3 client.4475.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:00.030 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.043+0000 7f298ffff640 3 client.4475.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:00.030 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.043+0000 7f298ffff640 20 client.4475.objecter dump_active .. 0 homeless 2024-03-29T11:15:00.030 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.043+0000 7f29a6ffd640 10 monclient: get_auth_request con 0x7f299406dc60 auth_method 0 2024-03-29T11:15:00.030 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.043+0000 7f298ffff640 10 client.4475.objecter ms_handle_connect 0x7f299406dc60 2024-03-29T11:15:00.030 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.051+0000 7f298ffff640 1 -- 172.21.15.18:0/2699189283 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f299c050580 con 0x7f29a80b17d0 2024-03-29T11:15:00.030 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.051+0000 7f298ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:00.031 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.051+0000 7f298ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:00.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.360+0000 7f29ade33640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:00.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.361+0000 7f29ade33640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:00.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.361+0000 7f29ade33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:15:00.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.361+0000 7f29ade33640 1 -- 172.21.15.18:0/2699189283 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2988005330 con 0x7f29a80b17d0 2024-03-29T11:15:00.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.362+0000 7f298ffff640 1 -- 172.21.15.18:0/2699189283 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f299c0524b0 con 0x7f29a80b17d0 2024-03-29T11:15:00.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.362+0000 7f298ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:00.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.362+0000 7f298ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:00.342 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:15:00.342 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:00.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.365+0000 7f29ade33640 10 librados: watch_flush enter 2024-03-29T11:15:00.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.366+0000 7f29ade33640 10 librados: watch_flush exit 2024-03-29T11:15:00.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.366+0000 7f29ade33640 20 client.4475.objecter shutdown clearing up homeless session... 2024-03-29T11:15:00.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.366+0000 7f29ade33640 10 client.4475.objecter successfully canceled tick 2024-03-29T11:15:00.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.366+0000 7f29ade33640 1 -- 172.21.15.18:0/2699189283 >> v1:172.21.15.70:6816/3101871562 conn(0x7f299406dc60 legacy=0x7f2994070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:00.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.366+0000 7f29ade33640 10 monclient: shutdown 2024-03-29T11:15:00.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.366+0000 7f29ade33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:00.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.366+0000 7f29ade33640 1 -- 172.21.15.18:0/2699189283 >> v1:172.21.15.18:6790/0 conn(0x7f29a80b17d0 legacy=0x7f29a8132320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:00.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.366+0000 7f29ade33640 1 -- 172.21.15.18:0/2699189283 shutdown_connections 2024-03-29T11:15:00.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.366+0000 7f29ade33640 1 -- 172.21.15.18:0/2699189283 >> 172.21.15.18:0/2699189283 conn(0x7f29a8066a00 msgr2=0x7f29a813d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:00.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.366+0000 7f29ade33640 1 -- 172.21.15.18:0/2699189283 shutdown_connections 2024-03-29T11:15:00.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.366+0000 7f29ade33640 1 -- 172.21.15.18:0/2699189283 wait complete. 2024-03-29T11:15:00.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:00.366+0000 7f29ade33640 1 librados: shutdown 2024-03-29T11:15:00.362 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:07.170 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:07.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.340+0000 7fc5f196c640 10 monclient: get_monmap_and_config 2024-03-29T11:15:07.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.340+0000 7fc5f196c640 10 monclient: build_initial_monmap 2024-03-29T11:15:07.318 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.340+0000 7fc5f196c640 10 monclient: monmap: 2024-03-29T11:15:07.318 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:15:07.318 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:07.318 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:15:07.341574+0000 2024-03-29T11:15:07.319 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:15:07.341574+0000 2024-03-29T11:15:07.319 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:07.319 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:07.319 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:15:07.319 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:15:07.319 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:15:07.319 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:07.321 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.344+0000 7fc5f196c640 1 Processor -- start 2024-03-29T11:15:07.322 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.344+0000 7fc5f196c640 1 -- start start 2024-03-29T11:15:07.322 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.344+0000 7fc5f196c640 10 monclient: init 2024-03-29T11:15:07.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.345+0000 7fc5f196c640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:07.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.345+0000 7fc5f196c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:15:07.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.345+0000 7fc5f196c640 10 monclient(hunting): picked mon.noname-a con 0x7fc5ec0a9750 addr v1:172.21.15.18:6789/0 2024-03-29T11:15:07.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.345+0000 7fc5f196c640 10 monclient(hunting): picked mon.noname-c con 0x7fc5ec149890 addr v1:172.21.15.70:6789/0 2024-03-29T11:15:07.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.345+0000 7fc5f196c640 10 monclient(hunting): picked mon.noname-b con 0x7fc5ec0afa30 addr v1:172.21.15.18:6790/0 2024-03-29T11:15:07.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.345+0000 7fc5f196c640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5ec0b00c0 con 0x7fc5ec0a9750 2024-03-29T11:15:07.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.345+0000 7fc5f196c640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5ec1464a0 con 0x7fc5ec149890 2024-03-29T11:15:07.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.346+0000 7fc5f196c640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5ec0ab280 con 0x7fc5ec0afa30 2024-03-29T11:15:07.325 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.346+0000 7fc5f196c640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:07.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.346+0000 7fc5f196c640 10 monclient(hunting): authenticate will time out at 1133.617932s 2024-03-29T11:15:07.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.346+0000 7fc5eb7fe640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fc5ec0afa30 0x7fc5ec151c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:44742/0 (socket says 172.21.15.18:44742) 2024-03-29T11:15:07.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.346+0000 7fc5eb7fe640 1 -- 172.21.15.18:0/2251692968 learned_addr learned my addr 172.21.15.18:0/2251692968 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:15:07.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.347+0000 7fc5e9ffb640 1 -- 172.21.15.18:0/2251692968 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2837767074 0 0) 0x7fc5ec0ab280 con 0x7fc5ec0afa30 2024-03-29T11:15:07.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.347+0000 7fc5e9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:15:07.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.347+0000 7fc5e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:07.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.347+0000 7fc5e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:07.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.347+0000 7fc5e9ffb640 10 monclient(hunting): my global_id is 4478 2024-03-29T11:15:07.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.347+0000 7fc5e9ffb640 1 -- 172.21.15.18:0/2251692968 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5d4003520 con 0x7fc5ec0afa30 2024-03-29T11:15:07.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.347+0000 7fc5e9ffb640 1 -- 172.21.15.18:0/2251692968 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1927543115 0 0) 0x7fc5d4003520 con 0x7fc5ec0afa30 2024-03-29T11:15:07.326 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.347+0000 7fc5e9ffb640 1 -- 172.21.15.18:0/2251692968 >> v1:172.21.15.70:6789/0 conn(0x7fc5ec149890 legacy=0x7fc5ec149da0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:15:07.327 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.350+0000 7fc5e9ffb640 1 -- 172.21.15.18:0/2251692968 >> v1:172.21.15.18:6789/0 conn(0x7fc5ec0a9750 legacy=0x7fc5ec1469a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:15:07.327 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.350+0000 7fc5e9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:15:07.328 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.350+0000 7fc5e9ffb640 1 monclient: found mon.noname-b 2024-03-29T11:15:07.328 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.350+0000 7fc5e9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:07.328 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.350+0000 7fc5e9ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:15:07.328 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.350+0000 7fc5e9ffb640 1 -- 172.21.15.18:0/2251692968 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5ec1470b0 con 0x7fc5ec0afa30 2024-03-29T11:15:07.328 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.350+0000 7fc5e9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:15:07.329 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.350+0000 7fc5e9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:15:07.329 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.350+0000 7fc5e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:07.329 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.350+0000 7fc5e9ffb640 1 -- 172.21.15.18:0/2251692968 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fc5e4001a10 con 0x7fc5ec0afa30 2024-03-29T11:15:07.329 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.350+0000 7fc5e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:07.329 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.350+0000 7fc5e9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:15:07.329 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.350+0000 7fc5e9ffb640 10 monclient: dump: 2024-03-29T11:15:07.329 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:15:07.329 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:07.329 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:07.329 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:07.329 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:07.329 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:07.329 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:15:07.330 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:15:07.330 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:15:07.330 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:07.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.350+0000 7fc5f196c640 5 monclient: authenticate success, global_id 4478 2024-03-29T11:15:07.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.350+0000 7fc5f196c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:07.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.350+0000 7fc5e9ffb640 1 -- 172.21.15.18:0/2251692968 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5e4001be0 con 0x7fc5ec0afa30 2024-03-29T11:15:07.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.350+0000 7fc5e9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:07.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.350+0000 7fc5f196c640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:07.331 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.350+0000 7fc5f196c640 4 set_mon_vals no callback set 2024-03-29T11:15:07.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.351+0000 7fc5e9ffb640 1 -- 172.21.15.18:0/2251692968 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fc5e4002030 con 0x7fc5ec0afa30 2024-03-29T11:15:07.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.351+0000 7fc5e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:07.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.351+0000 7fc5e9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:07.333 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.351+0000 7fc5e9ffb640 10 monclient: dump: 2024-03-29T11:15:07.334 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:15:07.334 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:07.334 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:07.334 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:07.334 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:07.334 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:07.334 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:15:07.334 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:15:07.334 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:15:07.335 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:07.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.351+0000 7fc5f196c640 10 monclient: shutdown 2024-03-29T11:15:07.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.351+0000 7fc5f196c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:07.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.351+0000 7fc5f196c640 1 -- 172.21.15.18:0/2251692968 >> v1:172.21.15.18:6790/0 conn(0x7fc5ec0afa30 legacy=0x7fc5ec151c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:07.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.351+0000 7fc5f196c640 1 -- 172.21.15.18:0/2251692968 shutdown_connections 2024-03-29T11:15:07.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.351+0000 7fc5f196c640 1 -- 172.21.15.18:0/2251692968 >> 172.21.15.18:0/2251692968 conn(0x7fc5ec0b3a00 msgr2=0x7fc5ec0b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:07.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.351+0000 7fc5f196c640 1 -- 172.21.15.18:0/2251692968 shutdown_connections 2024-03-29T11:15:07.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.351+0000 7fc5f196c640 1 -- 172.21.15.18:0/2251692968 wait complete. 2024-03-29T11:15:07.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.352+0000 7fc5f196c640 10 monclient: build_initial_monmap 2024-03-29T11:15:07.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.352+0000 7fc5f196c640 10 monclient: monmap: 2024-03-29T11:15:07.335 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:15:07.336 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:07.336 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:15:07.336 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:15:07.336 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:07.336 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:07.336 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:15:07.336 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:15:07.336 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:15:07.336 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:07.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.353+0000 7fc5f196c640 1 librados: starting msgr at 2024-03-29T11:15:07.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.353+0000 7fc5f196c640 1 librados: starting objecter 2024-03-29T11:15:07.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.353+0000 7fc5f196c640 1 Processor -- start 2024-03-29T11:15:07.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.353+0000 7fc5f196c640 1 -- start start 2024-03-29T11:15:07.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.353+0000 7fc5f196c640 1 librados: setting wanted keys 2024-03-29T11:15:07.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.353+0000 7fc5f196c640 1 librados: calling monclient init 2024-03-29T11:15:07.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.353+0000 7fc5f196c640 10 monclient: init 2024-03-29T11:15:07.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.354+0000 7fc5f196c640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:07.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.354+0000 7fc5f196c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:15:07.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.354+0000 7fc5f196c640 10 monclient(hunting): picked mon.noname-c con 0x7fc5ec0a9750 addr v1:172.21.15.18:6790/0 2024-03-29T11:15:07.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.355+0000 7fc5eaffd640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fc5ec0a9750 0x7fc5ec140090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:49336/0 (socket says 172.21.15.18:49336) 2024-03-29T11:15:07.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.356+0000 7fc5f196c640 10 monclient(hunting): picked mon.noname-a con 0x7fc5ec0afa30 addr v1:172.21.15.18:6789/0 2024-03-29T11:15:07.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.356+0000 7fc5f196c640 10 monclient(hunting): picked mon.noname-b con 0x7fc5ec149890 addr v1:172.21.15.70:6789/0 2024-03-29T11:15:07.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.356+0000 7fc5f196c640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5ec072180 con 0x7fc5ec0afa30 2024-03-29T11:15:07.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.356+0000 7fc5f196c640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5ec06fd70 con 0x7fc5ec149890 2024-03-29T11:15:07.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.356+0000 7fc5f196c640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5ec070010 con 0x7fc5ec0a9750 2024-03-29T11:15:07.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.356+0000 7fc5f196c640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:07.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.356+0000 7fc5f196c640 10 monclient(hunting): authenticate will time out at 1133.628027s 2024-03-29T11:15:07.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.356+0000 7fc5eaffd640 1 -- 172.21.15.18:0/1857926680 learned_addr learned my addr 172.21.15.18:0/1857926680 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:15:07.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.358+0000 7fc5d3fff640 10 client.?.objecter ms_handle_connect 0x7fc5ec0a9750 2024-03-29T11:15:07.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.359+0000 7fc5d3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:07.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.359+0000 7fc5d3fff640 1 -- 172.21.15.18:0/1857926680 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3889866411 0 0) 0x7fc5ec070010 con 0x7fc5ec0a9750 2024-03-29T11:15:07.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.359+0000 7fc5d3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:15:07.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.359+0000 7fc5d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:07.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.359+0000 7fc5d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:07.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.359+0000 7fc5d3fff640 10 monclient(hunting): my global_id is 4481 2024-03-29T11:15:07.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.359+0000 7fc5d3fff640 1 -- 172.21.15.18:0/1857926680 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5d8004eb0 con 0x7fc5ec0a9750 2024-03-29T11:15:07.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.359+0000 7fc5d3fff640 1 -- 172.21.15.18:0/1857926680 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 383127223 0 0) 0x7fc5d8004eb0 con 0x7fc5ec0a9750 2024-03-29T11:15:07.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.359+0000 7fc5d3fff640 1 -- 172.21.15.18:0/1857926680 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5ec070010 con 0x7fc5ec0a9750 2024-03-29T11:15:07.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.359+0000 7fc5d3fff640 1 -- 172.21.15.18:0/1857926680 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fc5dc001960 con 0x7fc5ec0a9750 2024-03-29T11:15:07.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.359+0000 7fc5d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:07.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.359+0000 7fc5d3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:15:07.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.359+0000 7fc5d3fff640 10 monclient(hunting): dump: 2024-03-29T11:15:07.343 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:15:07.343 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:07.343 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:07.343 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:07.343 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:07.343 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:07.343 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:15:07.343 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:15:07.343 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:15:07.343 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:07.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.359+0000 7fc5d3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:07.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.360+0000 7fc5d3fff640 1 -- 172.21.15.18:0/1857926680 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2862498740 0 0) 0x7fc5ec070010 con 0x7fc5ec0a9750 2024-03-29T11:15:07.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.360+0000 7fc5d3fff640 1 -- 172.21.15.18:0/1857926680 >> v1:172.21.15.70:6789/0 conn(0x7fc5ec149890 legacy=0x7fc5ec1dede0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:15:07.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.361+0000 7fc5d3fff640 1 -- 172.21.15.18:0/1857926680 >> v1:172.21.15.18:6789/0 conn(0x7fc5ec0afa30 legacy=0x7fc5ec1de6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:07.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.361+0000 7fc5d3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:15:07.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.361+0000 7fc5d3fff640 1 monclient: found mon.c 2024-03-29T11:15:07.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.361+0000 7fc5d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:07.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.361+0000 7fc5d3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:15:07.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.361+0000 7fc5d3fff640 1 -- 172.21.15.18:0/1857926680 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5ec0711e0 con 0x7fc5ec0a9750 2024-03-29T11:15:07.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.361+0000 7fc5d3fff640 10 monclient: _finish_auth 0 2024-03-29T11:15:07.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.361+0000 7fc5d3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:15:07.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.361+0000 7fc5d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:07.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.361+0000 7fc5f196c640 5 monclient: authenticate success, global_id 4481 2024-03-29T11:15:07.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.361+0000 7fc5f196c640 10 monclient: _renew_subs 2024-03-29T11:15:07.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.361+0000 7fc5f196c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:15:07.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.361+0000 7fc5f196c640 1 -- 172.21.15.18:0/1857926680 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5ec0727f0 con 0x7fc5ec0a9750 2024-03-29T11:15:07.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.361+0000 7fc5f196c640 10 client.4481.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:07.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.361+0000 7fc5f196c640 10 monclient: _renew_subs 2024-03-29T11:15:07.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.361+0000 7fc5f196c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:15:07.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.361+0000 7fc5f196c640 1 -- 172.21.15.18:0/1857926680 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5ec140e50 con 0x7fc5ec0a9750 2024-03-29T11:15:07.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.361+0000 7fc5f196c640 1 librados: init done 2024-03-29T11:15:07.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.361+0000 7fc5d3fff640 10 client.4481.objecter ms_handle_connect 0x7fc5ec0afa30 2024-03-29T11:15:07.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.361+0000 7fc5d3fff640 10 client.4481.objecter resend_mon_ops 2024-03-29T11:15:07.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.361+0000 7fc5d3fff640 1 -- 172.21.15.18:0/1857926680 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5dc001c80 con 0x7fc5ec0a9750 2024-03-29T11:15:07.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.361+0000 7fc5d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:07.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.361+0000 7fc5d3fff640 1 -- 172.21.15.18:0/1857926680 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fc5dc003a40 con 0x7fc5ec0a9750 2024-03-29T11:15:07.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.361+0000 7fc5d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:07.345 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.361+0000 7fc5d3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:07.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.361+0000 7fc5d3fff640 10 monclient: dump: 2024-03-29T11:15:07.346 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:15:07.346 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:07.346 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:07.346 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:07.346 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:07.346 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:07.346 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:15:07.346 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:15:07.346 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:15:07.346 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:07.346 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.361+0000 7fc5e97fa640 4 set_mon_vals no callback set 2024-03-29T11:15:07.348 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.362+0000 7fc5f196c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:07.348 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.362+0000 7fc5f196c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:07.348 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.362+0000 7fc5f196c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:15:07.348 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.362+0000 7fc5f196c640 1 -- 172.21.15.18:0/1857926680 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5cc005080 con 0x7fc5ec0a9750 2024-03-29T11:15:07.348 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.364+0000 7fc5d3fff640 1 -- 172.21.15.18:0/1857926680 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7fc5dc003ba0 con 0x7fc5ec0a9750 2024-03-29T11:15:07.348 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.365+0000 7fc5d3fff640 1 -- 172.21.15.18:0/1857926680 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7fc5dc084a70 con 0x7fc5ec0a9750 2024-03-29T11:15:07.348 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.365+0000 7fc5d3fff640 10 client.4481.objecter ms_dispatch 0x7fc5ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:07.348 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.365+0000 7fc5d3fff640 3 client.4481.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:07.348 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.365+0000 7fc5d3fff640 3 client.4481.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:07.348 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.365+0000 7fc5d3fff640 20 client.4481.objecter dump_active .. 0 homeless 2024-03-29T11:15:07.349 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.365+0000 7fc5ea7fc640 10 monclient: get_auth_request con 0x7fc5d806db60 auth_method 0 2024-03-29T11:15:07.349 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.366+0000 7fc5d3fff640 10 client.4481.objecter ms_handle_connect 0x7fc5d806db60 2024-03-29T11:15:07.349 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.371+0000 7fc5d3fff640 1 -- 172.21.15.18:0/1857926680 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc5dc050350 con 0x7fc5ec0a9750 2024-03-29T11:15:07.349 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.371+0000 7fc5d3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:07.349 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.371+0000 7fc5d3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:07.698 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.720+0000 7fc5f196c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:07.698 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.720+0000 7fc5f196c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:07.698 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.720+0000 7fc5f196c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:15:07.699 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.720+0000 7fc5f196c640 1 -- 172.21.15.18:0/1857926680 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5cc005330 con 0x7fc5ec0a9750 2024-03-29T11:15:07.700 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.722+0000 7fc5d3fff640 1 -- 172.21.15.18:0/1857926680 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7fc5dc052280 con 0x7fc5ec0a9750 2024-03-29T11:15:07.700 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.722+0000 7fc5d3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:07.700 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.722+0000 7fc5d3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:07.700 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:15:07.700 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:07.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.726+0000 7fc5f196c640 10 librados: watch_flush enter 2024-03-29T11:15:07.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.726+0000 7fc5f196c640 10 librados: watch_flush exit 2024-03-29T11:15:07.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.726+0000 7fc5f196c640 20 client.4481.objecter shutdown clearing up homeless session... 2024-03-29T11:15:07.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.726+0000 7fc5f196c640 10 client.4481.objecter successfully canceled tick 2024-03-29T11:15:07.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.726+0000 7fc5f196c640 1 -- 172.21.15.18:0/1857926680 >> v1:172.21.15.70:6816/3101871562 conn(0x7fc5d806db60 legacy=0x7fc5d8070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:07.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.726+0000 7fc5f196c640 10 monclient: shutdown 2024-03-29T11:15:07.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.726+0000 7fc5f196c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:07.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.726+0000 7fc5f196c640 1 -- 172.21.15.18:0/1857926680 >> v1:172.21.15.18:6790/0 conn(0x7fc5ec0a9750 legacy=0x7fc5ec140090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:07.705 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.727+0000 7fc5f196c640 1 -- 172.21.15.18:0/1857926680 shutdown_connections 2024-03-29T11:15:07.705 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.727+0000 7fc5f196c640 1 -- 172.21.15.18:0/1857926680 >> 172.21.15.18:0/1857926680 conn(0x7fc5ec0b3a00 msgr2=0x7fc5ec0b1150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:07.706 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.728+0000 7fc5f196c640 1 -- 172.21.15.18:0/1857926680 shutdown_connections 2024-03-29T11:15:07.706 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.728+0000 7fc5f196c640 1 -- 172.21.15.18:0/1857926680 wait complete. 2024-03-29T11:15:07.706 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:07.728+0000 7fc5f196c640 1 librados: shutdown 2024-03-29T11:15:07.723 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:14.530 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:14.686 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.707+0000 7f91051cf640 10 monclient: get_monmap_and_config 2024-03-29T11:15:14.687 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.707+0000 7f91051cf640 10 monclient: build_initial_monmap 2024-03-29T11:15:14.687 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.707+0000 7f91051cf640 10 monclient: monmap: 2024-03-29T11:15:14.687 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:15:14.687 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:14.687 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:15:14.709073+0000 2024-03-29T11:15:14.687 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:15:14.709073+0000 2024-03-29T11:15:14.687 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:14.687 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:14.687 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:15:14.687 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:15:14.687 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:15:14.687 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:14.690 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.711+0000 7f91051cf640 1 Processor -- start 2024-03-29T11:15:14.690 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.711+0000 7f91051cf640 1 -- start start 2024-03-29T11:15:14.690 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.712+0000 7f91051cf640 10 monclient: init 2024-03-29T11:15:14.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.712+0000 7f91051cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:14.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.712+0000 7f91051cf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:15:14.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.712+0000 7f91051cf640 10 monclient(hunting): picked mon.noname-c con 0x7f91000a9ac0 addr v1:172.21.15.70:6789/0 2024-03-29T11:15:14.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.713+0000 7f91051cf640 10 monclient(hunting): picked mon.noname-b con 0x7f910013ee90 addr v1:172.21.15.18:6790/0 2024-03-29T11:15:14.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.713+0000 7f91051cf640 10 monclient(hunting): picked mon.noname-a con 0x7f91000af900 addr v1:172.21.15.18:6789/0 2024-03-29T11:15:14.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.713+0000 7f91051cf640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91000afd10 con 0x7f91000af900 2024-03-29T11:15:14.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.713+0000 7f91051cf640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91000af6f0 con 0x7f91000a9ac0 2024-03-29T11:15:14.691 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.713+0000 7f91051cf640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91000aac80 con 0x7f910013ee90 2024-03-29T11:15:14.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.713+0000 7f91051cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:14.692 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.713+0000 7f91051cf640 10 monclient(hunting): authenticate will time out at 1140.985991s 2024-03-29T11:15:14.693 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.714+0000 7f90fe59c640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f910013ee90 0x7f910013b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:53720/0 (socket says 172.21.15.18:53720) 2024-03-29T11:15:14.693 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.714+0000 7f90fe59c640 1 -- 172.21.15.18:0/3264039887 learned_addr learned my addr 172.21.15.18:0/3264039887 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:15:14.694 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.715+0000 7f90fdd9b640 1 -- 172.21.15.18:0/3264039887 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2733913833 0 0) 0x7f91000aac80 con 0x7f910013ee90 2024-03-29T11:15:14.694 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.715+0000 7f90fdd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:15:14.694 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.715+0000 7f90fdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:14.694 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.715+0000 7f90fdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:14.694 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.715+0000 7f90fdd9b640 10 monclient(hunting): my global_id is 4484 2024-03-29T11:15:14.694 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.715+0000 7f90fdd9b640 1 -- 172.21.15.18:0/3264039887 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90e8003520 con 0x7f910013ee90 2024-03-29T11:15:14.694 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.716+0000 7f90fdd9b640 1 -- 172.21.15.18:0/3264039887 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 243832865 0 0) 0x7f90e8003520 con 0x7f910013ee90 2024-03-29T11:15:14.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.716+0000 7f90fdd9b640 1 -- 172.21.15.18:0/3264039887 >> v1:172.21.15.70:6789/0 conn(0x7f91000a9ac0 legacy=0x7f91000adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:15:14.695 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.716+0000 7f90fdd9b640 1 -- 172.21.15.18:0/3264039887 >> v1:172.21.15.18:6789/0 conn(0x7f91000af900 legacy=0x7f9100149b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:15:14.697 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.718+0000 7f90fdd9b640 10 monclient: _finish_hunting 0 2024-03-29T11:15:14.697 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.718+0000 7f90fdd9b640 1 monclient: found mon.noname-b 2024-03-29T11:15:14.697 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.718+0000 7f90fdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:14.697 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.718+0000 7f90fdd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:15:14.697 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.718+0000 7f90fdd9b640 1 -- 172.21.15.18:0/3264039887 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91000aaf60 con 0x7f910013ee90 2024-03-29T11:15:14.697 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.718+0000 7f90fdd9b640 10 monclient: _finish_auth 0 2024-03-29T11:15:14.697 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.718+0000 7f90fdd9b640 10 monclient: _check_auth_tickets 2024-03-29T11:15:14.697 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.718+0000 7f90fdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:14.697 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.718+0000 7f90fdd9b640 1 -- 172.21.15.18:0/3264039887 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f90f8001a10 con 0x7f910013ee90 2024-03-29T11:15:14.697 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.718+0000 7f90fdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:14.697 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.718+0000 7f90fdd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:15:14.698 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.718+0000 7f90fdd9b640 10 monclient: dump: 2024-03-29T11:15:14.698 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:15:14.698 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:14.698 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:14.698 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:14.698 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:14.698 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:14.698 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:15:14.698 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:15:14.698 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:15:14.698 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:14.698 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.718+0000 7f91051cf640 5 monclient: authenticate success, global_id 4484 2024-03-29T11:15:14.698 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.718+0000 7f91051cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:14.699 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.719+0000 7f90fdd9b640 1 -- 172.21.15.18:0/3264039887 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90f8001ca0 con 0x7f910013ee90 2024-03-29T11:15:14.699 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.720+0000 7f90fdd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:14.699 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.720+0000 7f91051cf640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:14.699 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.720+0000 7f91051cf640 4 set_mon_vals no callback set 2024-03-29T11:15:14.700 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.720+0000 7f90fdd9b640 1 -- 172.21.15.18:0/3264039887 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f90f8002030 con 0x7f910013ee90 2024-03-29T11:15:14.700 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.720+0000 7f90fdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:14.700 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.720+0000 7f90fdd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:14.700 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.720+0000 7f90fdd9b640 10 monclient: dump: 2024-03-29T11:15:14.700 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:15:14.700 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:14.700 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:14.700 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:14.700 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:14.700 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:14.700 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:15:14.700 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:15:14.700 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:15:14.700 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:14.700 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.720+0000 7f91051cf640 10 monclient: shutdown 2024-03-29T11:15:14.701 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.720+0000 7f91051cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:14.701 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.720+0000 7f91051cf640 1 -- 172.21.15.18:0/3264039887 >> v1:172.21.15.18:6790/0 conn(0x7f910013ee90 legacy=0x7f910013b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:14.701 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.721+0000 7f91051cf640 1 -- 172.21.15.18:0/3264039887 shutdown_connections 2024-03-29T11:15:14.701 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.721+0000 7f91051cf640 1 -- 172.21.15.18:0/3264039887 >> 172.21.15.18:0/3264039887 conn(0x7f91000b35e0 msgr2=0x7f91000b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:14.701 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.721+0000 7f91051cf640 1 -- 172.21.15.18:0/3264039887 shutdown_connections 2024-03-29T11:15:14.701 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.721+0000 7f91051cf640 1 -- 172.21.15.18:0/3264039887 wait complete. 2024-03-29T11:15:14.701 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.722+0000 7f91051cf640 10 monclient: build_initial_monmap 2024-03-29T11:15:14.701 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.722+0000 7f91051cf640 10 monclient: monmap: 2024-03-29T11:15:14.701 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:15:14.701 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:14.701 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:15:14.702 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:15:14.702 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:14.702 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:14.702 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:15:14.702 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:15:14.702 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:15:14.702 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:14.702 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.722+0000 7f91051cf640 1 librados: starting msgr at 2024-03-29T11:15:14.702 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.722+0000 7f91051cf640 1 librados: starting objecter 2024-03-29T11:15:14.703 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.723+0000 7f91051cf640 1 Processor -- start 2024-03-29T11:15:14.703 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.723+0000 7f91051cf640 1 -- start start 2024-03-29T11:15:14.703 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.723+0000 7f91051cf640 1 librados: setting wanted keys 2024-03-29T11:15:14.703 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.723+0000 7f91051cf640 1 librados: calling monclient init 2024-03-29T11:15:14.703 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.723+0000 7f91051cf640 10 monclient: init 2024-03-29T11:15:14.703 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.724+0000 7f91051cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:14.703 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.724+0000 7f91051cf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:15:14.703 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.724+0000 7f91051cf640 10 monclient(hunting): picked mon.noname-b con 0x7f91000a9ac0 addr v1:172.21.15.70:6789/0 2024-03-29T11:15:14.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.725+0000 7f91051cf640 10 monclient(hunting): picked mon.noname-c con 0x7f91000af900 addr v1:172.21.15.18:6790/0 2024-03-29T11:15:14.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.725+0000 7f91051cf640 10 monclient(hunting): picked mon.noname-a con 0x7f910013ee90 addr v1:172.21.15.18:6789/0 2024-03-29T11:15:14.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.725+0000 7f91051cf640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91000723f0 con 0x7f910013ee90 2024-03-29T11:15:14.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.725+0000 7f91051cf640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91000720f0 con 0x7f91000a9ac0 2024-03-29T11:15:14.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.725+0000 7f91051cf640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91001e1d90 con 0x7f91000af900 2024-03-29T11:15:14.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.725+0000 7f91051cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:14.704 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.725+0000 7f91051cf640 10 monclient(hunting): authenticate will time out at 1140.998861s 2024-03-29T11:15:14.707 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.728+0000 7f90fe59c640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f91000af900 0x7f91001e0f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:54974/0 (socket says 172.21.15.18:54974) 2024-03-29T11:15:14.707 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.728+0000 7f90fe59c640 1 -- 172.21.15.18:0/670166293 learned_addr learned my addr 172.21.15.18:0/670166293 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:15:14.709 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.728+0000 7f90e77fe640 10 client.?.objecter ms_handle_connect 0x7f91000af900 2024-03-29T11:15:14.709 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.728+0000 7f90e77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:14.709 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.728+0000 7f90e77fe640 10 client.?.objecter ms_handle_connect 0x7f910013ee90 2024-03-29T11:15:14.709 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.728+0000 7f90e77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:14.709 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.728+0000 7f90e77fe640 1 -- 172.21.15.18:0/670166293 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3276703433 0 0) 0x7f91001e1d90 con 0x7f91000af900 2024-03-29T11:15:14.709 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.728+0000 7f90e77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:15:14.710 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.728+0000 7f90e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:14.710 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.728+0000 7f90e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:14.710 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.728+0000 7f90e77fe640 10 monclient(hunting): my global_id is 4487 2024-03-29T11:15:14.710 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.728+0000 7f90e77fe640 1 -- 172.21.15.18:0/670166293 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90ec004eb0 con 0x7f91000af900 2024-03-29T11:15:14.710 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.728+0000 7f90e77fe640 1 -- 172.21.15.18:0/670166293 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3674264085 0 0) 0x7f91000723f0 con 0x7f910013ee90 2024-03-29T11:15:14.710 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.728+0000 7f90e77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:15:14.710 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.728+0000 7f90e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:14.710 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.728+0000 7f90e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:14.710 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.728+0000 7f90e77fe640 10 monclient(hunting): my global_id is 4425 2024-03-29T11:15:14.710 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.728+0000 7f90e77fe640 1 -- 172.21.15.18:0/670166293 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91001e1d90 con 0x7f910013ee90 2024-03-29T11:15:14.710 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.729+0000 7f90e77fe640 1 -- 172.21.15.18:0/670166293 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3804682925 0 0) 0x7f91001e1d90 con 0x7f910013ee90 2024-03-29T11:15:14.710 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.729+0000 7f90e77fe640 1 -- 172.21.15.18:0/670166293 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91000723f0 con 0x7f910013ee90 2024-03-29T11:15:14.710 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.729+0000 7f90e77fe640 1 -- 172.21.15.18:0/670166293 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f90f4001730 con 0x7f910013ee90 2024-03-29T11:15:14.710 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.729+0000 7f90e77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:14.710 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.729+0000 7f90e77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:15:14.711 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.729+0000 7f90e77fe640 10 monclient(hunting): dump: 2024-03-29T11:15:14.711 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:15:14.711 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:14.711 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:14.711 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:14.711 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:14.711 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:14.711 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:15:14.711 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:15:14.711 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:15:14.711 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:14.711 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.729+0000 7f90e77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:14.711 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.729+0000 7f90e77fe640 1 -- 172.21.15.18:0/670166293 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3146693479 0 0) 0x7f91000723f0 con 0x7f910013ee90 2024-03-29T11:15:14.711 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.729+0000 7f90e77fe640 1 -- 172.21.15.18:0/670166293 >> v1:172.21.15.18:6790/0 conn(0x7f91000af900 legacy=0x7f91001e0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:14.711 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.730+0000 7f90e77fe640 1 -- 172.21.15.18:0/670166293 >> v1:172.21.15.70:6789/0 conn(0x7f91000a9ac0 legacy=0x7f91001e7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:14.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.730+0000 7f90e77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:15:14.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.730+0000 7f90e77fe640 1 monclient: found mon.a 2024-03-29T11:15:14.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.730+0000 7f90e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:14.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.730+0000 7f90e77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:15:14.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.730+0000 7f90e77fe640 1 -- 172.21.15.18:0/670166293 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91000717e0 con 0x7f910013ee90 2024-03-29T11:15:14.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.730+0000 7f90e77fe640 10 monclient: _finish_auth 0 2024-03-29T11:15:14.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.730+0000 7f90e77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:15:14.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.730+0000 7f90e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:14.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.730+0000 7f90e77fe640 1 -- 172.21.15.18:0/670166293 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90f4001890 con 0x7f910013ee90 2024-03-29T11:15:14.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.730+0000 7f90e77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:14.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.730+0000 7f90e77fe640 1 -- 172.21.15.18:0/670166293 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f90f4003790 con 0x7f910013ee90 2024-03-29T11:15:14.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.730+0000 7f90e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:14.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.730+0000 7f90e77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:14.712 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.730+0000 7f90e77fe640 10 monclient: dump: 2024-03-29T11:15:14.712 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:15:14.713 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:14.713 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:14.713 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:14.713 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:14.713 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:14.713 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:15:14.713 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:15:14.713 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:15:14.713 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:14.713 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.730+0000 7f90fcd99640 4 set_mon_vals no callback set 2024-03-29T11:15:14.715 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.731+0000 7f91051cf640 5 monclient: authenticate success, global_id 4425 2024-03-29T11:15:14.715 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.731+0000 7f91051cf640 10 monclient: _renew_subs 2024-03-29T11:15:14.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.731+0000 7f91051cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:15:14.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.731+0000 7f91051cf640 1 -- 172.21.15.18:0/670166293 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91001e7900 con 0x7f910013ee90 2024-03-29T11:15:14.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.731+0000 7f91051cf640 10 client.4425.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:14.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.731+0000 7f91051cf640 10 monclient: _renew_subs 2024-03-29T11:15:14.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.731+0000 7f91051cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:15:14.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.731+0000 7f91051cf640 1 -- 172.21.15.18:0/670166293 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f91001c77a0 con 0x7f910013ee90 2024-03-29T11:15:14.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.731+0000 7f91051cf640 1 librados: init done 2024-03-29T11:15:14.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.733+0000 7f90e57fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:14.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.733+0000 7f90e57fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:14.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.733+0000 7f90e57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:15:14.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.733+0000 7f90e57fa640 1 -- 172.21.15.18:0/670166293 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91001e7ca0 con 0x7f910013ee90 2024-03-29T11:15:14.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.734+0000 7f90e77fe640 1 -- 172.21.15.18:0/670166293 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f90f4019760 con 0x7f910013ee90 2024-03-29T11:15:14.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.735+0000 7f90e77fe640 1 -- 172.21.15.18:0/670166293 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f90f4083720 con 0x7f910013ee90 2024-03-29T11:15:14.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.735+0000 7f90fed9d640 10 monclient: get_auth_request con 0x7f90ec06dce0 auth_method 0 2024-03-29T11:15:14.716 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.735+0000 7f90e77fe640 10 client.4425.objecter ms_dispatch 0x7f9100000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:14.717 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.735+0000 7f90e77fe640 3 client.4425.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:14.717 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.735+0000 7f90e77fe640 3 client.4425.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:14.717 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.735+0000 7f90e77fe640 20 client.4425.objecter dump_active .. 0 homeless 2024-03-29T11:15:14.717 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.735+0000 7f90e77fe640 10 client.4425.objecter ms_handle_connect 0x7f90ec06dce0 2024-03-29T11:15:14.721 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.743+0000 7f90e77fe640 1 -- 172.21.15.18:0/670166293 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f90f404f000 con 0x7f910013ee90 2024-03-29T11:15:14.721 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.743+0000 7f90e77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:14.722 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:14.743+0000 7f90e77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:15.041 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:15.063+0000 7f90e57fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:15.042 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:15.063+0000 7f90e57fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:15.042 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:15.063+0000 7f90e57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:15:15.042 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:15.063+0000 7f90e57fa640 1 -- 172.21.15.18:0/670166293 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f91001e7a90 con 0x7f910013ee90 2024-03-29T11:15:15.042 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:15.064+0000 7f90e77fe640 1 -- 172.21.15.18:0/670166293 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f90f4050f30 con 0x7f910013ee90 2024-03-29T11:15:15.042 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:15.064+0000 7f90e77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:15.042 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:15.064+0000 7f90e77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:15.043 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:15:15.043 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:15.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:15.067+0000 7f90e57fa640 10 librados: watch_flush enter 2024-03-29T11:15:15.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:15.067+0000 7f90e57fa640 10 librados: watch_flush exit 2024-03-29T11:15:15.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:15.067+0000 7f90e57fa640 20 client.4425.objecter shutdown clearing up homeless session... 2024-03-29T11:15:15.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:15.067+0000 7f90e57fa640 10 client.4425.objecter successfully canceled tick 2024-03-29T11:15:15.047 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:15.068+0000 7f90e57fa640 1 -- 172.21.15.18:0/670166293 >> v1:172.21.15.70:6816/3101871562 conn(0x7f90ec06dce0 legacy=0x7f90ec0701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:15.047 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:15.068+0000 7f90e57fa640 10 monclient: shutdown 2024-03-29T11:15:15.047 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:15.068+0000 7f90e57fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:15.047 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:15.068+0000 7f90e57fa640 1 -- 172.21.15.18:0/670166293 >> v1:172.21.15.18:6789/0 conn(0x7f910013ee90 legacy=0x7f91001e1680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:15.047 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:15.068+0000 7f90e57fa640 1 -- 172.21.15.18:0/670166293 shutdown_connections 2024-03-29T11:15:15.047 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:15.068+0000 7f90e57fa640 1 -- 172.21.15.18:0/670166293 >> 172.21.15.18:0/670166293 conn(0x7f91000b35e0 msgr2=0x7f91000b07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:15.047 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:15.068+0000 7f90e57fa640 1 -- 172.21.15.18:0/670166293 shutdown_connections 2024-03-29T11:15:15.047 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:15.068+0000 7f90e57fa640 1 -- 172.21.15.18:0/670166293 wait complete. 2024-03-29T11:15:15.047 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:15.068+0000 7f90e57fa640 1 librados: shutdown 2024-03-29T11:15:15.064 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:21.871 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:22.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.057+0000 7fde02682640 10 monclient: get_monmap_and_config 2024-03-29T11:15:22.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.057+0000 7fde02682640 10 monclient: build_initial_monmap 2024-03-29T11:15:22.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.057+0000 7fde02682640 10 monclient: monmap: 2024-03-29T11:15:22.035 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:15:22.036 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:22.036 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:15:22.058841+0000 2024-03-29T11:15:22.036 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:15:22.058841+0000 2024-03-29T11:15:22.036 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:22.036 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:22.036 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:15:22.036 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:15:22.036 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:15:22.036 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:22.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.059+0000 7fde02682640 1 Processor -- start 2024-03-29T11:15:22.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.059+0000 7fde02682640 1 -- start start 2024-03-29T11:15:22.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.059+0000 7fde02682640 10 monclient: init 2024-03-29T11:15:22.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.060+0000 7fde02682640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:22.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.060+0000 7fde02682640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:15:22.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.060+0000 7fde02682640 10 monclient(hunting): picked mon.noname-b con 0x7fddf40389e0 addr v1:172.21.15.18:6790/0 2024-03-29T11:15:22.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.060+0000 7fde02682640 10 monclient(hunting): picked mon.noname-c con 0x7fddf4038dc0 addr v1:172.21.15.70:6789/0 2024-03-29T11:15:22.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.060+0000 7fde02682640 10 monclient(hunting): picked mon.noname-a con 0x7fddf4030660 addr v1:172.21.15.18:6789/0 2024-03-29T11:15:22.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.060+0000 7fde02682640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf40c54c0 con 0x7fddf4030660 2024-03-29T11:15:22.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.060+0000 7fde02682640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf4036430 con 0x7fddf4038dc0 2024-03-29T11:15:22.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.060+0000 7fde02682640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf4039840 con 0x7fddf40389e0 2024-03-29T11:15:22.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.060+0000 7fde02682640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:22.039 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.060+0000 7fde02682640 10 monclient(hunting): authenticate will time out at 1148.332118s 2024-03-29T11:15:22.042 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.065+0000 7fde00e7f640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fddf4038dc0 0x7fddf40cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.18:37103/0 (socket says 172.21.15.18:37103) 2024-03-29T11:15:22.042 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.065+0000 7fde00e7f640 1 -- 172.21.15.18:0/1576350546 learned_addr learned my addr 172.21.15.18:0/1576350546 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:15:22.043 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.065+0000 7fddf3fff640 1 -- 172.21.15.18:0/1576350546 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2662783739 0 0) 0x7fddf4036430 con 0x7fddf4038dc0 2024-03-29T11:15:22.043 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.065+0000 7fddf3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:15:22.043 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.065+0000 7fddf3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:22.043 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.065+0000 7fddf3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:22.044 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.065+0000 7fddf3fff640 10 monclient(hunting): my global_id is 4405 2024-03-29T11:15:22.044 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.065+0000 7fddf3fff640 1 -- 172.21.15.18:0/1576350546 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdde8003520 con 0x7fddf4038dc0 2024-03-29T11:15:22.045 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.068+0000 7fddf3fff640 1 -- 172.21.15.18:0/1576350546 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2020590884 0 0) 0x7fdde8003520 con 0x7fddf4038dc0 2024-03-29T11:15:22.045 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.068+0000 7fddf3fff640 1 -- 172.21.15.18:0/1576350546 >> v1:172.21.15.18:6790/0 conn(0x7fddf40389e0 legacy=0x7fddf4033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:15:22.045 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.068+0000 7fddf3fff640 1 -- 172.21.15.18:0/1576350546 >> v1:172.21.15.18:6789/0 conn(0x7fddf4030660 legacy=0x7fddf40d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:15:22.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.068+0000 7fddf3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:15:22.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.068+0000 7fddf3fff640 1 monclient: found mon.noname-c 2024-03-29T11:15:22.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.068+0000 7fddf3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:22.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.068+0000 7fddf3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.70:6789/0 2024-03-29T11:15:22.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.068+0000 7fddf3fff640 1 -- 172.21.15.18:0/1576350546 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf4035690 con 0x7fddf4038dc0 2024-03-29T11:15:22.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.068+0000 7fddf3fff640 10 monclient: _finish_auth 0 2024-03-29T11:15:22.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.068+0000 7fddf3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:15:22.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.068+0000 7fddf3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:22.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.068+0000 7fddf3fff640 1 -- 172.21.15.18:0/1576350546 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fddec001670 con 0x7fddf4038dc0 2024-03-29T11:15:22.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.068+0000 7fddf3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:22.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.068+0000 7fddf3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:15:22.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.068+0000 7fddf3fff640 10 monclient: dump: 2024-03-29T11:15:22.046 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:15:22.046 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:22.047 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:22.047 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:22.047 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:22.047 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:22.047 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:15:22.047 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:15:22.047 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:15:22.047 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:22.047 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.068+0000 7fddf3fff640 1 -- 172.21.15.18:0/1576350546 <== mon.1 v1:172.21.15.70:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fddec001840 con 0x7fddf4038dc0 2024-03-29T11:15:22.047 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.068+0000 7fddf3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:22.047 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.068+0000 7fddf3fff640 1 -- 172.21.15.18:0/1576350546 <== mon.1 v1:172.21.15.70:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fddec001c90 con 0x7fddf4038dc0 2024-03-29T11:15:22.047 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.068+0000 7fddf3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:22.047 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.068+0000 7fddf3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:15:22.047 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.068+0000 7fddf3fff640 10 monclient: dump: 2024-03-29T11:15:22.048 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:15:22.048 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:22.048 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:22.048 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:22.048 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:22.048 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:22.048 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:15:22.048 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:15:22.048 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:15:22.048 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:22.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.069+0000 7fde02682640 5 monclient: authenticate success, global_id 4405 2024-03-29T11:15:22.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.069+0000 7fde02682640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:22.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.069+0000 7fde02682640 4 set_mon_vals no callback set 2024-03-29T11:15:22.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.070+0000 7fde02682640 10 monclient: shutdown 2024-03-29T11:15:22.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.070+0000 7fde02682640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:22.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.070+0000 7fde02682640 1 -- 172.21.15.18:0/1576350546 >> v1:172.21.15.70:6789/0 conn(0x7fddf4038dc0 legacy=0x7fddf40cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:22.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.070+0000 7fde02682640 1 -- 172.21.15.18:0/1576350546 shutdown_connections 2024-03-29T11:15:22.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.070+0000 7fde02682640 1 -- 172.21.15.18:0/1576350546 >> 172.21.15.18:0/1576350546 conn(0x7fddf4005c00 msgr2=0x7fddf4006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:22.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.071+0000 7fde02682640 1 -- 172.21.15.18:0/1576350546 shutdown_connections 2024-03-29T11:15:22.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.071+0000 7fde02682640 1 -- 172.21.15.18:0/1576350546 wait complete. 2024-03-29T11:15:22.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.071+0000 7fde02682640 10 monclient: build_initial_monmap 2024-03-29T11:15:22.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.071+0000 7fde02682640 10 monclient: monmap: 2024-03-29T11:15:22.051 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:15:22.051 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:22.051 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:15:22.051 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:15:22.051 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:22.051 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:22.051 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:15:22.051 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:15:22.051 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:15:22.052 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:22.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.072+0000 7fde02682640 1 librados: starting msgr at 2024-03-29T11:15:22.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.072+0000 7fde02682640 1 librados: starting objecter 2024-03-29T11:15:22.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.072+0000 7fde02682640 1 Processor -- start 2024-03-29T11:15:22.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.072+0000 7fde02682640 1 -- start start 2024-03-29T11:15:22.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.072+0000 7fde02682640 1 librados: setting wanted keys 2024-03-29T11:15:22.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.072+0000 7fde02682640 1 librados: calling monclient init 2024-03-29T11:15:22.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.072+0000 7fde02682640 10 monclient: init 2024-03-29T11:15:22.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.073+0000 7fde02682640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:22.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.073+0000 7fde02682640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:15:22.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.073+0000 7fde02682640 10 monclient(hunting): picked mon.noname-b con 0x7fddf4030660 addr v1:172.21.15.70:6789/0 2024-03-29T11:15:22.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.073+0000 7fde02682640 10 monclient(hunting): picked mon.noname-c con 0x7fddf40389e0 addr v1:172.21.15.18:6790/0 2024-03-29T11:15:22.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.073+0000 7fde02682640 10 monclient(hunting): picked mon.noname-a con 0x7fddf402d150 addr v1:172.21.15.18:6789/0 2024-03-29T11:15:22.054 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.073+0000 7fde02682640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf40c7ff0 con 0x7fddf402d150 2024-03-29T11:15:22.054 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.073+0000 7fde02682640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf402e4d0 con 0x7fddf4030660 2024-03-29T11:15:22.054 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.073+0000 7fde02682640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf40c7420 con 0x7fddf40389e0 2024-03-29T11:15:22.054 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.073+0000 7fde02682640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:22.054 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.073+0000 7fde02682640 10 monclient(hunting): authenticate will time out at 1148.345653s 2024-03-29T11:15:22.057 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.080+0000 7fde01e81640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fddf402d150 0x7fddf4173e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:60909/0 (socket says 172.21.15.18:60909) 2024-03-29T11:15:22.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.080+0000 7fde01e81640 1 -- 172.21.15.18:0/3059348025 learned_addr learned my addr 172.21.15.18:0/3059348025 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:15:22.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.080+0000 7fddf1ffb640 10 client.?.objecter ms_handle_connect 0x7fddf402d150 2024-03-29T11:15:22.061 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.081+0000 7fddf1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:22.061 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.081+0000 7fddf1ffb640 10 client.?.objecter ms_handle_connect 0x7fddf40389e0 2024-03-29T11:15:22.061 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.081+0000 7fddf1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:22.061 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.081+0000 7fddf1ffb640 1 -- 172.21.15.18:0/3059348025 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 641465368 0 0) 0x7fddf40c7ff0 con 0x7fddf402d150 2024-03-29T11:15:22.062 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.081+0000 7fddf1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:15:22.062 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.081+0000 7fddf1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:22.062 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.081+0000 7fddf1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:22.062 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.081+0000 7fddf1ffb640 10 monclient(hunting): my global_id is 4428 2024-03-29T11:15:22.062 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.081+0000 7fddf1ffb640 1 -- 172.21.15.18:0/3059348025 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdde0004eb0 con 0x7fddf402d150 2024-03-29T11:15:22.062 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.081+0000 7fddf1ffb640 1 -- 172.21.15.18:0/3059348025 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2569634168 0 0) 0x7fddf40c7420 con 0x7fddf40389e0 2024-03-29T11:15:22.062 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.081+0000 7fddf1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:15:22.062 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.081+0000 7fddf1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:22.062 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.082+0000 7fddf1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:22.062 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.082+0000 7fddf1ffb640 10 monclient(hunting): my global_id is 4490 2024-03-29T11:15:22.062 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.082+0000 7fddf1ffb640 1 -- 172.21.15.18:0/3059348025 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddf40c7ff0 con 0x7fddf40389e0 2024-03-29T11:15:22.062 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.082+0000 7fddf1ffb640 1 -- 172.21.15.18:0/3059348025 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 5020472 0 0) 0x7fdde0004eb0 con 0x7fddf402d150 2024-03-29T11:15:22.062 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.082+0000 7fddf1ffb640 1 -- 172.21.15.18:0/3059348025 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fddf40c7420 con 0x7fddf402d150 2024-03-29T11:15:22.063 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.082+0000 7fddf1ffb640 1 -- 172.21.15.18:0/3059348025 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fddfc04fdd0 con 0x7fddf402d150 2024-03-29T11:15:22.063 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.082+0000 7fddf1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:22.063 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.082+0000 7fddf1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:15:22.063 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.082+0000 7fddf1ffb640 10 monclient(hunting): dump: 2024-03-29T11:15:22.063 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:15:22.063 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:22.063 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:22.063 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:22.063 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:22.063 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:22.063 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:15:22.064 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:15:22.064 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:15:22.064 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:22.064 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.082+0000 7fddf1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:22.064 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.082+0000 7fddf1ffb640 1 -- 172.21.15.18:0/3059348025 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3808507909 0 0) 0x7fddf40c7420 con 0x7fddf402d150 2024-03-29T11:15:22.064 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.082+0000 7fddf1ffb640 1 -- 172.21.15.18:0/3059348025 >> v1:172.21.15.18:6790/0 conn(0x7fddf40389e0 legacy=0x7fddf4169b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:22.064 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.082+0000 7fddf1ffb640 1 -- 172.21.15.18:0/3059348025 >> v1:172.21.15.70:6789/0 conn(0x7fddf4030660 legacy=0x7fddf4169400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:22.064 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.082+0000 7fddf1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:15:22.064 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.083+0000 7fddf1ffb640 1 monclient: found mon.a 2024-03-29T11:15:22.064 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.083+0000 7fddf1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:22.064 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.083+0000 7fddf1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:15:22.064 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.083+0000 7fddf1ffb640 1 -- 172.21.15.18:0/3059348025 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf40ca8e0 con 0x7fddf402d150 2024-03-29T11:15:22.064 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.083+0000 7fddf1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:15:22.064 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.083+0000 7fddf1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:15:22.065 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.083+0000 7fddf1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:22.066 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.083+0000 7fde02682640 5 monclient: authenticate success, global_id 4428 2024-03-29T11:15:22.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.083+0000 7fde02682640 10 monclient: _renew_subs 2024-03-29T11:15:22.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.083+0000 7fde02682640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:15:22.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.083+0000 7fde02682640 1 -- 172.21.15.18:0/3059348025 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddf40cab90 con 0x7fddf402d150 2024-03-29T11:15:22.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.083+0000 7fde02682640 10 client.4428.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:22.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.083+0000 7fde02682640 10 monclient: _renew_subs 2024-03-29T11:15:22.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.083+0000 7fde02682640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:15:22.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.083+0000 7fde02682640 1 -- 172.21.15.18:0/3059348025 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fddf4175740 con 0x7fddf402d150 2024-03-29T11:15:22.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.083+0000 7fde02682640 1 librados: init done 2024-03-29T11:15:22.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.084+0000 7fddf1ffb640 1 -- 172.21.15.18:0/3059348025 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fddfc04f400 con 0x7fddf402d150 2024-03-29T11:15:22.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.084+0000 7fddf1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:22.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.084+0000 7fddf1ffb640 1 -- 172.21.15.18:0/3059348025 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fddfc075710 con 0x7fddf402d150 2024-03-29T11:15:22.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.084+0000 7fddf1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:22.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.084+0000 7fddf1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:22.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.084+0000 7fddf1ffb640 10 monclient: dump: 2024-03-29T11:15:22.068 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:15:22.068 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:22.068 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:22.068 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:22.068 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:22.068 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:22.068 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:15:22.068 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:15:22.068 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:15:22.068 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:22.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.085+0000 7fddf37fe640 4 set_mon_vals no callback set 2024-03-29T11:15:22.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.086+0000 7fde02682640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:22.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.086+0000 7fde02682640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:22.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.086+0000 7fde02682640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:15:22.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.086+0000 7fde02682640 1 -- 172.21.15.18:0/3059348025 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdddc005080 con 0x7fddf402d150 2024-03-29T11:15:22.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.091+0000 7fddf1ffb640 1 -- 172.21.15.18:0/3059348025 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7fddfc067bf0 con 0x7fddf402d150 2024-03-29T11:15:22.070 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.091+0000 7fddf1ffb640 1 -- 172.21.15.18:0/3059348025 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7fddfc0ff4d0 con 0x7fddf402d150 2024-03-29T11:15:22.070 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.091+0000 7fddf1ffb640 10 client.4428.objecter ms_dispatch 0x7fddfc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:22.070 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.091+0000 7fddf1ffb640 3 client.4428.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:22.070 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.091+0000 7fddf1ffb640 3 client.4428.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:22.070 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.091+0000 7fde01680640 10 monclient: get_auth_request con 0x7fdde006db60 auth_method 0 2024-03-29T11:15:22.070 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.091+0000 7fddf1ffb640 20 client.4428.objecter dump_active .. 0 homeless 2024-03-29T11:15:22.070 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.092+0000 7fddf1ffb640 10 client.4428.objecter ms_handle_connect 0x7fdde006db60 2024-03-29T11:15:22.072 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.095+0000 7fddf1ffb640 1 -- 172.21.15.18:0/3059348025 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fddfc0cadb0 con 0x7fddf402d150 2024-03-29T11:15:22.073 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.095+0000 7fddf1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:22.073 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.095+0000 7fddf1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:22.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.443+0000 7fde02682640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:22.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.443+0000 7fde02682640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:22.420 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.443+0000 7fde02682640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:15:22.421 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.443+0000 7fde02682640 1 -- 172.21.15.18:0/3059348025 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdddc005330 con 0x7fddf402d150 2024-03-29T11:15:22.422 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.445+0000 7fddf1ffb640 1 -- 172.21.15.18:0/3059348025 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7fddfc0ccce0 con 0x7fddf402d150 2024-03-29T11:15:22.422 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.445+0000 7fddf1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:22.422 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.445+0000 7fddf1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:22.425 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:15:22.425 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:22.428 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.450+0000 7fde02682640 10 librados: watch_flush enter 2024-03-29T11:15:22.428 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.450+0000 7fde02682640 10 librados: watch_flush exit 2024-03-29T11:15:22.428 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.450+0000 7fde02682640 20 client.4428.objecter shutdown clearing up homeless session... 2024-03-29T11:15:22.428 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.450+0000 7fde02682640 10 client.4428.objecter successfully canceled tick 2024-03-29T11:15:22.428 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.451+0000 7fde02682640 1 -- 172.21.15.18:0/3059348025 >> v1:172.21.15.70:6816/3101871562 conn(0x7fdde006db60 legacy=0x7fdde0070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:22.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.451+0000 7fde02682640 10 monclient: shutdown 2024-03-29T11:15:22.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.451+0000 7fde02682640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:22.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.451+0000 7fde02682640 1 -- 172.21.15.18:0/3059348025 >> v1:172.21.15.18:6789/0 conn(0x7fddf402d150 legacy=0x7fddf4173e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:22.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.452+0000 7fde02682640 1 -- 172.21.15.18:0/3059348025 shutdown_connections 2024-03-29T11:15:22.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.452+0000 7fde02682640 1 -- 172.21.15.18:0/3059348025 >> 172.21.15.18:0/3059348025 conn(0x7fddf4005c00 msgr2=0x7fddf4034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:22.430 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.452+0000 7fde02682640 1 -- 172.21.15.18:0/3059348025 shutdown_connections 2024-03-29T11:15:22.430 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.452+0000 7fde02682640 1 -- 172.21.15.18:0/3059348025 wait complete. 2024-03-29T11:15:22.430 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:22.452+0000 7fde02682640 1 librados: shutdown 2024-03-29T11:15:22.447 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:29.254 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:29.415 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.434+0000 7f94a0f9f640 10 monclient: get_monmap_and_config 2024-03-29T11:15:29.415 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.434+0000 7f94a0f9f640 10 monclient: build_initial_monmap 2024-03-29T11:15:29.415 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.434+0000 7f94a0f9f640 10 monclient: monmap: 2024-03-29T11:15:29.415 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:15:29.415 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:29.415 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:15:29.435723+0000 2024-03-29T11:15:29.415 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:15:29.435723+0000 2024-03-29T11:15:29.415 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:29.415 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:29.415 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:15:29.415 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:15:29.416 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:15:29.416 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:29.416 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.437+0000 7f94a0f9f640 1 Processor -- start 2024-03-29T11:15:29.418 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.440+0000 7f94a0f9f640 1 -- start start 2024-03-29T11:15:29.418 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.440+0000 7f94a0f9f640 10 monclient: init 2024-03-29T11:15:29.419 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.441+0000 7f94a0f9f640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:29.419 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.441+0000 7f94a0f9f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:15:29.419 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.441+0000 7f94a0f9f640 10 monclient(hunting): picked mon.noname-c con 0x7f9494006410 addr v1:172.21.15.70:6789/0 2024-03-29T11:15:29.419 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.441+0000 7f94a0f9f640 10 monclient(hunting): picked mon.noname-b con 0x7f949402a640 addr v1:172.21.15.18:6790/0 2024-03-29T11:15:29.419 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.441+0000 7f94a0f9f640 10 monclient(hunting): picked mon.noname-a con 0x7f94940327d0 addr v1:172.21.15.18:6789/0 2024-03-29T11:15:29.419 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.441+0000 7f94a0f9f640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f949402c820 con 0x7f94940327d0 2024-03-29T11:15:29.419 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.441+0000 7f94a0f9f640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f949402a120 con 0x7f9494006410 2024-03-29T11:15:29.419 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.441+0000 7f94a0f9f640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94940c75b0 con 0x7f949402a640 2024-03-29T11:15:29.419 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.441+0000 7f94a0f9f640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:29.419 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.441+0000 7f94a0f9f640 10 monclient(hunting): authenticate will time out at 1155.713861s 2024-03-29T11:15:29.425 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.447+0000 7f949bfff640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f94940327d0 0x7f94940d1550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:49063/0 (socket says 172.21.15.18:49063) 2024-03-29T11:15:29.425 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.447+0000 7f949bfff640 1 -- 172.21.15.18:0/2056141285 learned_addr learned my addr 172.21.15.18:0/2056141285 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:15:29.428 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.448+0000 7f949a7fc640 1 -- 172.21.15.18:0/2056141285 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3698047841 0 0) 0x7f94940c75b0 con 0x7f949402a640 2024-03-29T11:15:29.428 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.448+0000 7f949a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:15:29.428 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.448+0000 7f949a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:29.428 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.448+0000 7f949a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:29.428 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.448+0000 7f949a7fc640 10 monclient(hunting): my global_id is 4493 2024-03-29T11:15:29.428 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.448+0000 7f949a7fc640 1 -- 172.21.15.18:0/2056141285 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f948c003520 con 0x7f949402a640 2024-03-29T11:15:29.428 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.448+0000 7f949a7fc640 1 -- 172.21.15.18:0/2056141285 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3695205398 0 0) 0x7f949402a120 con 0x7f9494006410 2024-03-29T11:15:29.428 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.449+0000 7f949a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:15:29.428 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.449+0000 7f949a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:29.428 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.449+0000 7f949a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:29.428 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.449+0000 7f949a7fc640 10 monclient(hunting): my global_id is 4408 2024-03-29T11:15:29.428 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.449+0000 7f949a7fc640 1 -- 172.21.15.18:0/2056141285 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94940c75b0 con 0x7f9494006410 2024-03-29T11:15:29.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.449+0000 7f949a7fc640 1 -- 172.21.15.18:0/2056141285 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 329356392 0 0) 0x7f948c003520 con 0x7f949402a640 2024-03-29T11:15:29.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.449+0000 7f949a7fc640 1 -- 172.21.15.18:0/2056141285 >> v1:172.21.15.70:6789/0 conn(0x7f9494006410 legacy=0x7f94940c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:29.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.449+0000 7f949a7fc640 1 -- 172.21.15.18:0/2056141285 >> v1:172.21.15.18:6789/0 conn(0x7f94940327d0 legacy=0x7f94940d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:29.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.449+0000 7f949a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:15:29.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.449+0000 7f949a7fc640 1 monclient: found mon.noname-b 2024-03-29T11:15:29.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.449+0000 7f949a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:29.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.449+0000 7f949a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:15:29.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.449+0000 7f949a7fc640 1 -- 172.21.15.18:0/2056141285 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9494033110 con 0x7f949402a640 2024-03-29T11:15:29.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.449+0000 7f949a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:15:29.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.449+0000 7f949a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:15:29.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.449+0000 7f949a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:29.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.449+0000 7f949a7fc640 1 -- 172.21.15.18:0/2056141285 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f949c04f730 con 0x7f949402a640 2024-03-29T11:15:29.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.449+0000 7f949a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:29.429 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.449+0000 7f949a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:15:29.430 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.449+0000 7f949a7fc640 10 monclient: dump: 2024-03-29T11:15:29.430 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:15:29.430 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:29.430 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:29.430 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:29.430 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:29.430 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:29.430 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:15:29.430 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:15:29.430 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:15:29.430 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:29.430 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.449+0000 7f94a0f9f640 5 monclient: authenticate success, global_id 4493 2024-03-29T11:15:29.430 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.449+0000 7f94a0f9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:29.430 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.450+0000 7f949a7fc640 1 -- 172.21.15.18:0/2056141285 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f949c0008d0 con 0x7f949402a640 2024-03-29T11:15:29.431 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.450+0000 7f949a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:29.431 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.450+0000 7f949a7fc640 1 -- 172.21.15.18:0/2056141285 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f949c0758b0 con 0x7f949402a640 2024-03-29T11:15:29.431 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.450+0000 7f949a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:29.431 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.450+0000 7f949a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:15:29.431 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.450+0000 7f949a7fc640 10 monclient: dump: 2024-03-29T11:15:29.431 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:15:29.431 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:29.431 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:29.431 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:29.431 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:29.431 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:29.431 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:15:29.431 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:15:29.431 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:15:29.431 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:29.431 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.450+0000 7f94a0f9f640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:29.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.450+0000 7f94a0f9f640 4 set_mon_vals no callback set 2024-03-29T11:15:29.432 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.451+0000 7f94a0f9f640 10 monclient: shutdown 2024-03-29T11:15:29.433 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.451+0000 7f94a0f9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:29.433 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.451+0000 7f94a0f9f640 1 -- 172.21.15.18:0/2056141285 >> v1:172.21.15.18:6790/0 conn(0x7f949402a640 legacy=0x7f94940320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:29.433 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.451+0000 7f94a0f9f640 1 -- 172.21.15.18:0/2056141285 shutdown_connections 2024-03-29T11:15:29.433 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.451+0000 7f94a0f9f640 1 -- 172.21.15.18:0/2056141285 >> 172.21.15.18:0/2056141285 conn(0x7f94940058f0 msgr2=0x7f9494005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:29.433 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.451+0000 7f94a0f9f640 1 -- 172.21.15.18:0/2056141285 shutdown_connections 2024-03-29T11:15:29.433 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.451+0000 7f94a0f9f640 1 -- 172.21.15.18:0/2056141285 wait complete. 2024-03-29T11:15:29.433 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.452+0000 7f94a0f9f640 10 monclient: build_initial_monmap 2024-03-29T11:15:29.433 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.452+0000 7f94a0f9f640 10 monclient: monmap: 2024-03-29T11:15:29.433 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:15:29.433 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:29.433 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:15:29.433 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:15:29.433 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:29.433 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:29.434 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:15:29.434 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:15:29.434 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:15:29.434 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:29.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.453+0000 7f94a0f9f640 1 librados: starting msgr at 2024-03-29T11:15:29.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.454+0000 7f94a0f9f640 1 librados: starting objecter 2024-03-29T11:15:29.434 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.454+0000 7f94a0f9f640 1 Processor -- start 2024-03-29T11:15:29.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.456+0000 7f94a0f9f640 1 -- start start 2024-03-29T11:15:29.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.456+0000 7f94a0f9f640 1 librados: setting wanted keys 2024-03-29T11:15:29.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.456+0000 7f94a0f9f640 1 librados: calling monclient init 2024-03-29T11:15:29.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.456+0000 7f94a0f9f640 10 monclient: init 2024-03-29T11:15:29.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.457+0000 7f94a0f9f640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:29.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.457+0000 7f94a0f9f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:15:29.435 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.457+0000 7f94a0f9f640 10 monclient(hunting): picked mon.noname-b con 0x7f9494006410 addr v1:172.21.15.70:6789/0 2024-03-29T11:15:29.438 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.460+0000 7f94a0f9f640 10 monclient(hunting): picked mon.noname-c con 0x7f949402a640 addr v1:172.21.15.18:6790/0 2024-03-29T11:15:29.438 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.460+0000 7f949affd640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f949402a640 0x7f94940bfd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:55377/0 (socket says 172.21.15.18:55377) 2024-03-29T11:15:29.438 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.460+0000 7f949affd640 1 -- 172.21.15.18:0/2304299827 learned_addr learned my addr 172.21.15.18:0/2304299827 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:15:29.438 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.460+0000 7f94a0f9f640 10 monclient(hunting): picked mon.noname-a con 0x7f94940327d0 addr v1:172.21.15.18:6789/0 2024-03-29T11:15:29.438 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.460+0000 7f94a0f9f640 1 -- 172.21.15.18:0/2304299827 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94940c1e80 con 0x7f94940327d0 2024-03-29T11:15:29.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.460+0000 7f94a0f9f640 1 -- 172.21.15.18:0/2304299827 --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94940befd0 con 0x7f9494006410 2024-03-29T11:15:29.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.460+0000 7f94a0f9f640 1 -- 172.21.15.18:0/2304299827 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94940bf160 con 0x7f949402a640 2024-03-29T11:15:29.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.460+0000 7f94a0f9f640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:29.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.460+0000 7f94a0f9f640 10 monclient(hunting): authenticate will time out at 1155.732861s 2024-03-29T11:15:29.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.467+0000 7f9487fff640 10 client.?.objecter ms_handle_connect 0x7f949402a640 2024-03-29T11:15:29.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.467+0000 7f9487fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:29.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.467+0000 7f9487fff640 10 client.?.objecter ms_handle_connect 0x7f9494006410 2024-03-29T11:15:29.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.467+0000 7f9487fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:29.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.467+0000 7f9487fff640 10 client.?.objecter ms_handle_connect 0x7f94940327d0 2024-03-29T11:15:29.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.467+0000 7f9487fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:29.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.467+0000 7f9487fff640 1 -- 172.21.15.18:0/2304299827 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3789694899 0 0) 0x7f94940bf160 con 0x7f949402a640 2024-03-29T11:15:29.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.467+0000 7f9487fff640 10 monclient(hunting): _negotiate 2024-03-29T11:15:29.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.467+0000 7f9487fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:29.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.467+0000 7f9487fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:29.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.467+0000 7f9487fff640 10 monclient(hunting): my global_id is 4496 2024-03-29T11:15:29.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.467+0000 7f9487fff640 1 -- 172.21.15.18:0/2304299827 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9480004eb0 con 0x7f949402a640 2024-03-29T11:15:29.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.467+0000 7f9487fff640 1 -- 172.21.15.18:0/2304299827 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2598820755 0 0) 0x7f94940befd0 con 0x7f9494006410 2024-03-29T11:15:29.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.467+0000 7f9487fff640 10 monclient(hunting): _negotiate 2024-03-29T11:15:29.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.467+0000 7f9487fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:29.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.467+0000 7f9487fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:29.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.467+0000 7f9487fff640 10 monclient(hunting): my global_id is 4411 2024-03-29T11:15:29.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.467+0000 7f9487fff640 1 -- 172.21.15.18:0/2304299827 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94940bf160 con 0x7f9494006410 2024-03-29T11:15:29.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.468+0000 7f9487fff640 1 -- 172.21.15.18:0/2304299827 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1151745279 0 0) 0x7f94940c1e80 con 0x7f94940327d0 2024-03-29T11:15:29.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.468+0000 7f9487fff640 10 monclient(hunting): _negotiate 2024-03-29T11:15:29.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.468+0000 7f9487fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:29.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.468+0000 7f9487fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:29.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.468+0000 7f9487fff640 10 monclient(hunting): my global_id is 4431 2024-03-29T11:15:29.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.468+0000 7f9487fff640 1 -- 172.21.15.18:0/2304299827 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94940befd0 con 0x7f94940327d0 2024-03-29T11:15:29.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.469+0000 7f9487fff640 1 -- 172.21.15.18:0/2304299827 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2856606706 0 0) 0x7f94940bf160 con 0x7f9494006410 2024-03-29T11:15:29.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.469+0000 7f9487fff640 1 -- 172.21.15.18:0/2304299827 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94940c1e80 con 0x7f9494006410 2024-03-29T11:15:29.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.469+0000 7f9487fff640 1 -- 172.21.15.18:0/2304299827 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f9490001810 con 0x7f9494006410 2024-03-29T11:15:29.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.469+0000 7f9487fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:29.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.469+0000 7f9487fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:15:29.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.469+0000 7f9487fff640 10 monclient(hunting): dump: 2024-03-29T11:15:29.449 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:15:29.449 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:29.449 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:29.449 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:29.449 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:29.449 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:29.449 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:15:29.449 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:15:29.450 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:15:29.450 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:29.450 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.469+0000 7f9487fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:29.450 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.469+0000 7f9487fff640 1 -- 172.21.15.18:0/2304299827 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2125299926 0 0) 0x7f94940befd0 con 0x7f94940327d0 2024-03-29T11:15:29.450 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.469+0000 7f9487fff640 1 -- 172.21.15.18:0/2304299827 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9480006a90 con 0x7f94940327d0 2024-03-29T11:15:29.450 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.469+0000 7f9487fff640 1 -- 172.21.15.18:0/2304299827 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f94880017a0 con 0x7f94940327d0 2024-03-29T11:15:29.450 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.469+0000 7f9487fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:29.450 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.469+0000 7f9487fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:29.450 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.469+0000 7f9487fff640 10 monclient(hunting): dump: 2024-03-29T11:15:29.450 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:15:29.450 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:29.450 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:29.450 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:29.450 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:29.451 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:29.451 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:15:29.451 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:15:29.451 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:15:29.451 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:29.453 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.470+0000 7f9487fff640 1 -- 172.21.15.18:0/2304299827 <== mon.1 v1:172.21.15.70:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2751814836 0 0) 0x7f94940c1e80 con 0x7f9494006410 2024-03-29T11:15:29.453 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.470+0000 7f9487fff640 1 -- 172.21.15.18:0/2304299827 >> v1:172.21.15.18:6790/0 conn(0x7f949402a640 legacy=0x7f94940bfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:29.453 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.470+0000 7f9487fff640 1 -- 172.21.15.18:0/2304299827 >> v1:172.21.15.18:6789/0 conn(0x7f94940327d0 legacy=0x7f94940aece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:29.453 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.470+0000 7f9487fff640 10 monclient: _finish_hunting 0 2024-03-29T11:15:29.453 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.470+0000 7f9487fff640 1 monclient: found mon.b 2024-03-29T11:15:29.453 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.470+0000 7f9487fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:29.453 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.470+0000 7f9487fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:15:29.453 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.470+0000 7f9487fff640 1 -- 172.21.15.18:0/2304299827 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94940ad470 con 0x7f9494006410 2024-03-29T11:15:29.453 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.470+0000 7f9487fff640 10 monclient: _finish_auth 0 2024-03-29T11:15:29.453 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.470+0000 7f9487fff640 10 monclient: _check_auth_tickets 2024-03-29T11:15:29.453 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.470+0000 7f9487fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:29.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.470+0000 7f94a0f9f640 5 monclient: authenticate success, global_id 4411 2024-03-29T11:15:29.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.470+0000 7f94a0f9f640 10 monclient: _renew_subs 2024-03-29T11:15:29.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.470+0000 7f94a0f9f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:15:29.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.470+0000 7f94a0f9f640 1 -- 172.21.15.18:0/2304299827 --> v1:172.21.15.70:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94940ad6e0 con 0x7f9494006410 2024-03-29T11:15:29.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.470+0000 7f94a0f9f640 10 client.4411.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:29.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.470+0000 7f94a0f9f640 10 monclient: _renew_subs 2024-03-29T11:15:29.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.470+0000 7f94a0f9f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:15:29.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.470+0000 7f94a0f9f640 1 -- 172.21.15.18:0/2304299827 --> v1:172.21.15.70:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f94940c4b10 con 0x7f9494006410 2024-03-29T11:15:29.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.470+0000 7f94a0f9f640 1 librados: init done 2024-03-29T11:15:29.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.471+0000 7f9487fff640 1 -- 172.21.15.18:0/2304299827 <== mon.1 v1:172.21.15.70:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9490001ca0 con 0x7f9494006410 2024-03-29T11:15:29.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.471+0000 7f9487fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:29.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.471+0000 7f9499ffb640 4 set_mon_vals no callback set 2024-03-29T11:15:29.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.473+0000 7f9487fff640 1 -- 172.21.15.18:0/2304299827 <== mon.1 v1:172.21.15.70:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f9490001810 con 0x7f9494006410 2024-03-29T11:15:29.454 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.473+0000 7f9487fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:29.455 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.474+0000 7f94a0f9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:29.455 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.474+0000 7f9487fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:15:29.455 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.474+0000 7f9487fff640 10 monclient: dump: 2024-03-29T11:15:29.455 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:15:29.455 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:29.455 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:29.455 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:29.455 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:29.455 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:29.455 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:15:29.455 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:15:29.455 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:15:29.455 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:29.455 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.474+0000 7f94a0f9f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:29.455 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.474+0000 7f9487fff640 1 -- 172.21.15.18:0/2304299827 <== mon.1 v1:172.21.15.70:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f9490019500 con 0x7f9494006410 2024-03-29T11:15:29.455 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.474+0000 7f94a0f9f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:15:29.456 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.474+0000 7f94a0f9f640 1 -- 172.21.15.18:0/2304299827 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f947c005080 con 0x7f9494006410 2024-03-29T11:15:29.456 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.475+0000 7f949affd640 10 monclient: get_auth_request con 0x7f948006e2c0 auth_method 0 2024-03-29T11:15:29.456 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.475+0000 7f9487fff640 1 -- 172.21.15.18:0/2304299827 <== mon.1 v1:172.21.15.70:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f9490083410 con 0x7f9494006410 2024-03-29T11:15:29.456 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.475+0000 7f9487fff640 10 client.4411.objecter ms_dispatch 0x7f949c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:29.456 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.475+0000 7f9487fff640 3 client.4411.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:29.456 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.475+0000 7f9487fff640 3 client.4411.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:29.456 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.475+0000 7f9487fff640 20 client.4411.objecter dump_active .. 0 homeless 2024-03-29T11:15:29.456 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.476+0000 7f9487fff640 10 client.4411.objecter ms_handle_connect 0x7f948006e2c0 2024-03-29T11:15:29.458 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.480+0000 7f9487fff640 1 -- 172.21.15.18:0/2304299827 <== mon.1 v1:172.21.15.70:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f949004ecf0 con 0x7f9494006410 2024-03-29T11:15:29.458 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.481+0000 7f9487fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:29.458 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.481+0000 7f9487fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:29.801 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.823+0000 7f94a0f9f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:29.801 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.823+0000 7f94a0f9f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:29.801 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.823+0000 7f94a0f9f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:15:29.801 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.823+0000 7f94a0f9f640 1 -- 172.21.15.18:0/2304299827 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f947c005330 con 0x7f9494006410 2024-03-29T11:15:29.801 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.824+0000 7f9487fff640 1 -- 172.21.15.18:0/2304299827 <== mon.1 v1:172.21.15.70:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f9490050c20 con 0x7f9494006410 2024-03-29T11:15:29.801 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.824+0000 7f9487fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:29.801 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.824+0000 7f9487fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:29.805 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:15:29.806 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:29.806 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.829+0000 7f94a0f9f640 10 librados: watch_flush enter 2024-03-29T11:15:29.806 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.829+0000 7f94a0f9f640 10 librados: watch_flush exit 2024-03-29T11:15:29.807 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.829+0000 7f94a0f9f640 20 client.4411.objecter shutdown clearing up homeless session... 2024-03-29T11:15:29.807 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.829+0000 7f94a0f9f640 10 client.4411.objecter successfully canceled tick 2024-03-29T11:15:29.808 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.830+0000 7f94a0f9f640 1 -- 172.21.15.18:0/2304299827 >> v1:172.21.15.70:6816/3101871562 conn(0x7f948006e2c0 legacy=0x7f9480070780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:29.808 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.831+0000 7f94a0f9f640 10 monclient: shutdown 2024-03-29T11:15:29.808 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.831+0000 7f94a0f9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:29.808 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.831+0000 7f94a0f9f640 1 -- 172.21.15.18:0/2304299827 >> v1:172.21.15.70:6789/0 conn(0x7f9494006410 legacy=0x7f94940c2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:29.808 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.831+0000 7f94a0f9f640 1 -- 172.21.15.18:0/2304299827 shutdown_connections 2024-03-29T11:15:29.808 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.831+0000 7f94a0f9f640 1 -- 172.21.15.18:0/2304299827 >> 172.21.15.18:0/2304299827 conn(0x7f94940058f0 msgr2=0x7f94940043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:29.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.831+0000 7f94a0f9f640 1 -- 172.21.15.18:0/2304299827 shutdown_connections 2024-03-29T11:15:29.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.831+0000 7f94a0f9f640 1 -- 172.21.15.18:0/2304299827 wait complete. 2024-03-29T11:15:29.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:29.831+0000 7f94a0f9f640 1 librados: shutdown 2024-03-29T11:15:29.830 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:36.637 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:36.788 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.810+0000 7f921a30b640 10 monclient: get_monmap_and_config 2024-03-29T11:15:36.788 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.810+0000 7f921a30b640 10 monclient: build_initial_monmap 2024-03-29T11:15:36.788 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.810+0000 7f921a30b640 10 monclient: monmap: 2024-03-29T11:15:36.788 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:15:36.788 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:36.788 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:15:36.811510+0000 2024-03-29T11:15:36.788 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:15:36.811510+0000 2024-03-29T11:15:36.788 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:36.788 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:36.789 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:15:36.789 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:15:36.789 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:15:36.789 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:36.789 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.811+0000 7f921a30b640 1 Processor -- start 2024-03-29T11:15:36.791 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.811+0000 7f921a30b640 1 -- start start 2024-03-29T11:15:36.791 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.811+0000 7f921a30b640 10 monclient: init 2024-03-29T11:15:36.791 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.812+0000 7f921a30b640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:36.791 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.812+0000 7f921a30b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:15:36.791 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.812+0000 7f921a30b640 10 monclient(hunting): picked mon.noname-c con 0x7f92140b4820 addr v1:172.21.15.70:6789/0 2024-03-29T11:15:36.791 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.812+0000 7f921a30b640 10 monclient(hunting): picked mon.noname-a con 0x7f92140a9ce0 addr v1:172.21.15.18:6789/0 2024-03-29T11:15:36.791 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.812+0000 7f921a30b640 10 monclient(hunting): picked mon.noname-b con 0x7f92140b3cd0 addr v1:172.21.15.18:6790/0 2024-03-29T11:15:36.791 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.812+0000 7f921a30b640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92140b4180 con 0x7f92140a9ce0 2024-03-29T11:15:36.791 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.812+0000 7f921a30b640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92140aaf60 con 0x7f92140b4820 2024-03-29T11:15:36.791 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.812+0000 7f921a30b640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f921413c200 con 0x7f92140b3cd0 2024-03-29T11:15:36.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.812+0000 7f921a30b640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:36.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.812+0000 7f921a30b640 10 monclient(hunting): authenticate will time out at 1163.084898s 2024-03-29T11:15:36.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.814+0000 7f9219b0a640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f92140b3cd0 0x7f921414d7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:47338/0 (socket says 172.21.15.18:47338) 2024-03-29T11:15:36.792 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.814+0000 7f9219b0a640 1 -- 172.21.15.18:0/2120828724 learned_addr learned my addr 172.21.15.18:0/2120828724 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:15:36.794 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.817+0000 7f920bfff640 1 -- 172.21.15.18:0/2120828724 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3114149000 0 0) 0x7f92140aaf60 con 0x7f92140b4820 2024-03-29T11:15:36.794 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.817+0000 7f920bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:15:36.795 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.817+0000 7f920bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:36.795 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.817+0000 7f920bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:36.795 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.817+0000 7f920bfff640 10 monclient(hunting): my global_id is 4414 2024-03-29T11:15:36.800 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.817+0000 7f920bfff640 1 -- 172.21.15.18:0/2120828724 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9200003520 con 0x7f92140b4820 2024-03-29T11:15:36.801 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.819+0000 7f920bfff640 1 -- 172.21.15.18:0/2120828724 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 498648957 0 0) 0x7f921413c200 con 0x7f92140b3cd0 2024-03-29T11:15:36.801 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.819+0000 7f920bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:15:36.801 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.819+0000 7f920bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:36.801 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.819+0000 7f920bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:36.801 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.820+0000 7f920bfff640 10 monclient(hunting): my global_id is 4499 2024-03-29T11:15:36.801 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.820+0000 7f920bfff640 1 -- 172.21.15.18:0/2120828724 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92140aaf60 con 0x7f92140b3cd0 2024-03-29T11:15:36.801 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.820+0000 7f920bfff640 1 -- 172.21.15.18:0/2120828724 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 91354880 0 0) 0x7f9200003520 con 0x7f92140b4820 2024-03-29T11:15:36.801 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.820+0000 7f920bfff640 1 -- 172.21.15.18:0/2120828724 >> v1:172.21.15.18:6790/0 conn(0x7f92140b3cd0 legacy=0x7f921414d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:36.801 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.820+0000 7f920bfff640 1 -- 172.21.15.18:0/2120828724 >> v1:172.21.15.18:6789/0 conn(0x7f92140a9ce0 legacy=0x7f9214145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:15:36.801 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.820+0000 7f920bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:15:36.801 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.820+0000 7f920bfff640 1 monclient: found mon.noname-c 2024-03-29T11:15:36.801 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.820+0000 7f920bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:36.801 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.820+0000 7f920bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.70:6789/0 2024-03-29T11:15:36.802 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.820+0000 7f920bfff640 1 -- 172.21.15.18:0/2120828724 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9214145e80 con 0x7f92140b4820 2024-03-29T11:15:36.802 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.820+0000 7f920bfff640 10 monclient: _finish_auth 0 2024-03-29T11:15:36.802 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.820+0000 7f920bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:15:36.802 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.820+0000 7f920bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:36.802 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.820+0000 7f921a30b640 5 monclient: authenticate success, global_id 4414 2024-03-29T11:15:36.802 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.820+0000 7f921a30b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:36.802 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.821+0000 7f920bfff640 1 -- 172.21.15.18:0/2120828724 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f9210001690 con 0x7f92140b4820 2024-03-29T11:15:36.802 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.821+0000 7f920bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:36.802 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.821+0000 7f920bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:15:36.802 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.821+0000 7f920bfff640 10 monclient: dump: 2024-03-29T11:15:36.802 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:15:36.802 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:36.802 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:36.803 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:36.803 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:36.803 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:36.803 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:15:36.803 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:15:36.803 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:15:36.803 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:36.803 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.821+0000 7f921a30b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:36.803 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.821+0000 7f920bfff640 1 -- 172.21.15.18:0/2120828724 <== mon.1 v1:172.21.15.70:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92100018b0 con 0x7f92140b4820 2024-03-29T11:15:36.803 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.821+0000 7f920bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:36.803 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.821+0000 7f921a30b640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:36.803 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.821+0000 7f921a30b640 4 set_mon_vals no callback set 2024-03-29T11:15:36.803 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.821+0000 7f920bfff640 1 -- 172.21.15.18:0/2120828724 <== mon.1 v1:172.21.15.70:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f9210001c40 con 0x7f92140b4820 2024-03-29T11:15:36.803 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.821+0000 7f920bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:36.803 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.821+0000 7f920bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:15:36.804 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.821+0000 7f920bfff640 10 monclient: dump: 2024-03-29T11:15:36.804 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:15:36.804 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:36.804 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:36.804 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:36.804 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:36.804 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:36.804 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:15:36.804 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:15:36.804 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:15:36.804 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:36.804 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.822+0000 7f921a30b640 10 monclient: shutdown 2024-03-29T11:15:36.804 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.822+0000 7f921a30b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:36.804 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.822+0000 7f921a30b640 1 -- 172.21.15.18:0/2120828724 >> v1:172.21.15.70:6789/0 conn(0x7f92140b4820 legacy=0x7f921413baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:36.805 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.822+0000 7f921a30b640 1 -- 172.21.15.18:0/2120828724 shutdown_connections 2024-03-29T11:15:36.805 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.823+0000 7f921a30b640 1 -- 172.21.15.18:0/2120828724 >> 172.21.15.18:0/2120828724 conn(0x7f921406d070 msgr2=0x7f9214066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:36.805 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.823+0000 7f921a30b640 1 -- 172.21.15.18:0/2120828724 shutdown_connections 2024-03-29T11:15:36.807 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.823+0000 7f921a30b640 1 -- 172.21.15.18:0/2120828724 wait complete. 2024-03-29T11:15:36.807 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.824+0000 7f921a30b640 10 monclient: build_initial_monmap 2024-03-29T11:15:36.807 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.824+0000 7f921a30b640 10 monclient: monmap: 2024-03-29T11:15:36.807 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:15:36.807 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:36.807 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:15:36.807 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:15:36.807 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:36.808 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:36.808 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:15:36.808 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:15:36.808 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:15:36.808 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:36.808 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.825+0000 7f921a30b640 1 librados: starting msgr at 2024-03-29T11:15:36.808 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.825+0000 7f921a30b640 1 librados: starting objecter 2024-03-29T11:15:36.808 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.825+0000 7f921a30b640 1 Processor -- start 2024-03-29T11:15:36.808 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.826+0000 7f921a30b640 1 -- start start 2024-03-29T11:15:36.808 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.827+0000 7f921a30b640 1 librados: setting wanted keys 2024-03-29T11:15:36.808 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.827+0000 7f921a30b640 1 librados: calling monclient init 2024-03-29T11:15:36.808 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.827+0000 7f921a30b640 10 monclient: init 2024-03-29T11:15:36.808 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.827+0000 7f921a30b640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:36.808 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.827+0000 7f921a30b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:15:36.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.827+0000 7f921a30b640 10 monclient(hunting): picked mon.noname-c con 0x7f92140a9ce0 addr v1:172.21.15.18:6790/0 2024-03-29T11:15:36.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.828+0000 7f9219309640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f92140a9ce0 0x7f9214143200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:48672/0 (socket says 172.21.15.18:48672) 2024-03-29T11:15:36.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.828+0000 7f9219309640 1 -- 172.21.15.18:0/1160660344 learned_addr learned my addr 172.21.15.18:0/1160660344 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:15:36.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.829+0000 7f921a30b640 10 monclient(hunting): picked mon.noname-a con 0x7f92140b3cd0 addr v1:172.21.15.18:6789/0 2024-03-29T11:15:36.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.829+0000 7f9209ffb640 10 client.?.objecter ms_handle_connect 0x7f92140a9ce0 2024-03-29T11:15:36.809 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.829+0000 7f9209ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:36.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.829+0000 7f921a30b640 10 monclient(hunting): picked mon.noname-b con 0x7f92140b4820 addr v1:172.21.15.70:6789/0 2024-03-29T11:15:36.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.831+0000 7f921a30b640 1 -- 172.21.15.18:0/1160660344 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f921413a6e0 con 0x7f92140b3cd0 2024-03-29T11:15:36.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.831+0000 7f921a30b640 1 -- 172.21.15.18:0/1160660344 --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f921413b510 con 0x7f92140b4820 2024-03-29T11:15:36.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.831+0000 7f921a30b640 1 -- 172.21.15.18:0/1160660344 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f921413ba60 con 0x7f92140a9ce0 2024-03-29T11:15:36.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.831+0000 7f921a30b640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:36.810 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.831+0000 7f921a30b640 10 monclient(hunting): authenticate will time out at 1163.103156s 2024-03-29T11:15:36.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.832+0000 7f9209ffb640 10 client.?.objecter ms_handle_connect 0x7f92140b3cd0 2024-03-29T11:15:36.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.832+0000 7f9209ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:36.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.833+0000 7f9209ffb640 1 -- 172.21.15.18:0/1160660344 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3056647419 0 0) 0x7f921413a6e0 con 0x7f92140b3cd0 2024-03-29T11:15:36.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.833+0000 7f9209ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:15:36.812 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.833+0000 7f9209ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:36.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.833+0000 7f9209ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:36.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.833+0000 7f9209ffb640 10 monclient(hunting): my global_id is 4434 2024-03-29T11:15:36.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.833+0000 7f9209ffb640 1 -- 172.21.15.18:0/1160660344 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91f8004eb0 con 0x7f92140b3cd0 2024-03-29T11:15:36.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.834+0000 7f9209ffb640 10 client.?.objecter ms_handle_connect 0x7f92140b4820 2024-03-29T11:15:36.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.834+0000 7f9209ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:36.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.834+0000 7f9209ffb640 1 -- 172.21.15.18:0/1160660344 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3168036330 0 0) 0x7f921413b510 con 0x7f92140b4820 2024-03-29T11:15:36.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.834+0000 7f9209ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:15:36.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.834+0000 7f9209ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:36.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.834+0000 7f9209ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:36.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.834+0000 7f9209ffb640 10 monclient(hunting): my global_id is 4417 2024-03-29T11:15:36.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.834+0000 7f9209ffb640 1 -- 172.21.15.18:0/1160660344 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f921413a6e0 con 0x7f92140b4820 2024-03-29T11:15:36.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.834+0000 7f9209ffb640 1 -- 172.21.15.18:0/1160660344 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 442450300 0 0) 0x7f91f8004eb0 con 0x7f92140b3cd0 2024-03-29T11:15:36.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.834+0000 7f9209ffb640 1 -- 172.21.15.18:0/1160660344 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f921413b510 con 0x7f92140b3cd0 2024-03-29T11:15:36.813 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.834+0000 7f9209ffb640 1 -- 172.21.15.18:0/1160660344 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f920c001bf0 con 0x7f92140b3cd0 2024-03-29T11:15:36.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.834+0000 7f9209ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:36.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.834+0000 7f9209ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:15:36.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.834+0000 7f9209ffb640 10 monclient(hunting): dump: 2024-03-29T11:15:36.814 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:15:36.814 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:36.814 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:36.814 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:36.814 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:36.814 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:36.814 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:15:36.814 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:15:36.814 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:15:36.814 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:36.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.834+0000 7f9209ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:36.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.835+0000 7f9209ffb640 1 -- 172.21.15.18:0/1160660344 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 628749971 0 0) 0x7f921413ba60 con 0x7f92140a9ce0 2024-03-29T11:15:36.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.835+0000 7f9209ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:15:36.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.835+0000 7f9209ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:36.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.835+0000 7f9209ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:36.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.835+0000 7f9209ffb640 10 monclient(hunting): my global_id is 4502 2024-03-29T11:15:36.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.835+0000 7f9209ffb640 1 -- 172.21.15.18:0/1160660344 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91f8004eb0 con 0x7f92140a9ce0 2024-03-29T11:15:36.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.835+0000 7f9209ffb640 1 -- 172.21.15.18:0/1160660344 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4264124170 0 0) 0x7f921413b510 con 0x7f92140b3cd0 2024-03-29T11:15:36.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.835+0000 7f9209ffb640 1 -- 172.21.15.18:0/1160660344 >> v1:172.21.15.18:6790/0 conn(0x7f92140a9ce0 legacy=0x7f9214143200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:36.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.835+0000 7f9209ffb640 1 -- 172.21.15.18:0/1160660344 >> v1:172.21.15.70:6789/0 conn(0x7f92140b4820 legacy=0x7f9214133060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:36.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.835+0000 7f9209ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:15:36.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.835+0000 7f9209ffb640 1 monclient: found mon.a 2024-03-29T11:15:36.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.835+0000 7f9209ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:36.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.835+0000 7f9209ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:15:36.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.835+0000 7f9209ffb640 1 -- 172.21.15.18:0/1160660344 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f921413b780 con 0x7f92140b3cd0 2024-03-29T11:15:36.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.835+0000 7f9209ffb640 10 monclient: _finish_auth 0 2024-03-29T11:15:36.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.835+0000 7f9209ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:15:36.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.835+0000 7f9209ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:36.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.835+0000 7f921a30b640 5 monclient: authenticate success, global_id 4434 2024-03-29T11:15:36.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.835+0000 7f921a30b640 10 monclient: _renew_subs 2024-03-29T11:15:36.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.835+0000 7f921a30b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:15:36.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.835+0000 7f921a30b640 1 -- 172.21.15.18:0/1160660344 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9214133770 con 0x7f92140b3cd0 2024-03-29T11:15:36.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.835+0000 7f921a30b640 10 client.4434.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:36.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.835+0000 7f921a30b640 10 monclient: _renew_subs 2024-03-29T11:15:36.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.836+0000 7f921a30b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:15:36.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.836+0000 7f921a30b640 1 -- 172.21.15.18:0/1160660344 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9214143910 con 0x7f92140b3cd0 2024-03-29T11:15:36.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.836+0000 7f921a30b640 1 librados: init done 2024-03-29T11:15:36.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.838+0000 7f9209ffb640 1 -- 172.21.15.18:0/1160660344 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f920c003970 con 0x7f92140b3cd0 2024-03-29T11:15:36.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.838+0000 7f9209ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:36.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.838+0000 7f9209ffb640 1 -- 172.21.15.18:0/1160660344 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f920c002100 con 0x7f92140b3cd0 2024-03-29T11:15:36.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.838+0000 7f9209ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:36.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.838+0000 7f9209ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:36.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.838+0000 7f9209ffb640 10 monclient: dump: 2024-03-29T11:15:36.818 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:15:36.818 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:36.818 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:36.818 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:36.818 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:36.818 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:36.818 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:15:36.818 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:15:36.818 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:15:36.819 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:36.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.839+0000 7f9209ffb640 1 -- 172.21.15.18:0/1160660344 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f920c019bb0 con 0x7f92140b3cd0 2024-03-29T11:15:36.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.839+0000 7f920b7fe640 4 set_mon_vals no callback set 2024-03-29T11:15:36.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.840+0000 7f9219309640 10 monclient: get_auth_request con 0x7f91f806ded0 auth_method 0 2024-03-29T11:15:36.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.841+0000 7f9209ffb640 1 -- 172.21.15.18:0/1160660344 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f920c083c90 con 0x7f92140b3cd0 2024-03-29T11:15:36.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.841+0000 7f9209ffb640 10 client.4434.objecter ms_dispatch 0x7f9214000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:36.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.841+0000 7f9209ffb640 3 client.4434.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:36.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.841+0000 7f9209ffb640 3 client.4434.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:36.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.841+0000 7f9209ffb640 20 client.4434.objecter dump_active .. 0 homeless 2024-03-29T11:15:36.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.841+0000 7f9209ffb640 10 client.4434.objecter ms_handle_connect 0x7f91f806ded0 2024-03-29T11:15:36.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.841+0000 7f921a30b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:36.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.841+0000 7f921a30b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:36.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.841+0000 7f921a30b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:15:36.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.841+0000 7f921a30b640 1 -- 172.21.15.18:0/1160660344 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91f4005080 con 0x7f92140b3cd0 2024-03-29T11:15:36.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.852+0000 7f9209ffb640 1 -- 172.21.15.18:0/1160660344 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f920c04f440 con 0x7f92140b3cd0 2024-03-29T11:15:36.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.852+0000 7f9209ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:36.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:36.852+0000 7f9209ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:37.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:37.212+0000 7f921a30b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:37.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:37.212+0000 7f921a30b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:37.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:37.212+0000 7f921a30b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:15:37.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:37.212+0000 7f921a30b640 1 -- 172.21.15.18:0/1160660344 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f91f4005330 con 0x7f92140b3cd0 2024-03-29T11:15:37.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:37.215+0000 7f9209ffb640 1 -- 172.21.15.18:0/1160660344 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f920c051540 con 0x7f92140b3cd0 2024-03-29T11:15:37.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:37.215+0000 7f9209ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:37.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:37.215+0000 7f9209ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:37.193 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:15:37.193 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:37.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:37.219+0000 7f921a30b640 10 librados: watch_flush enter 2024-03-29T11:15:37.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:37.219+0000 7f921a30b640 10 librados: watch_flush exit 2024-03-29T11:15:37.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:37.219+0000 7f921a30b640 20 client.4434.objecter shutdown clearing up homeless session... 2024-03-29T11:15:37.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:37.219+0000 7f921a30b640 10 client.4434.objecter successfully canceled tick 2024-03-29T11:15:37.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:37.221+0000 7f921a30b640 1 -- 172.21.15.18:0/1160660344 >> v1:172.21.15.70:6816/3101871562 conn(0x7f91f806ded0 legacy=0x7f91f8070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:37.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:37.221+0000 7f921a30b640 10 monclient: shutdown 2024-03-29T11:15:37.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:37.221+0000 7f921a30b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:37.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:37.221+0000 7f921a30b640 1 -- 172.21.15.18:0/1160660344 >> v1:172.21.15.18:6789/0 conn(0x7f92140b3cd0 legacy=0x7f9214132950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:37.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:37.221+0000 7f921a30b640 1 -- 172.21.15.18:0/1160660344 shutdown_connections 2024-03-29T11:15:37.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:37.221+0000 7f921a30b640 1 -- 172.21.15.18:0/1160660344 >> 172.21.15.18:0/1160660344 conn(0x7f921406d070 msgr2=0x7f92140b8220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:37.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:37.222+0000 7f921a30b640 1 -- 172.21.15.18:0/1160660344 shutdown_connections 2024-03-29T11:15:37.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:37.222+0000 7f921a30b640 1 -- 172.21.15.18:0/1160660344 wait complete. 2024-03-29T11:15:37.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:37.222+0000 7f921a30b640 1 librados: shutdown 2024-03-29T11:15:37.218 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:44.026 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:44.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.203+0000 7f44a1e0f640 10 monclient: get_monmap_and_config 2024-03-29T11:15:44.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.203+0000 7f44a1e0f640 10 monclient: build_initial_monmap 2024-03-29T11:15:44.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.203+0000 7f44a1e0f640 10 monclient: monmap: 2024-03-29T11:15:44.181 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:15:44.181 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:44.182 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:15:44.204448+0000 2024-03-29T11:15:44.182 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:15:44.204448+0000 2024-03-29T11:15:44.182 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:44.182 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:44.182 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:15:44.182 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:15:44.182 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:15:44.182 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:44.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.204+0000 7f44a1e0f640 1 Processor -- start 2024-03-29T11:15:44.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.204+0000 7f44a1e0f640 1 -- start start 2024-03-29T11:15:44.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.204+0000 7f44a1e0f640 10 monclient: init 2024-03-29T11:15:44.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.206+0000 7f44a1e0f640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:44.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.206+0000 7f44a1e0f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:15:44.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.206+0000 7f44a1e0f640 10 monclient(hunting): picked mon.noname-a con 0x7f449c071de0 addr v1:172.21.15.18:6789/0 2024-03-29T11:15:44.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.206+0000 7f44a1e0f640 10 monclient(hunting): picked mon.noname-c con 0x7f449c0721c0 addr v1:172.21.15.70:6789/0 2024-03-29T11:15:44.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.206+0000 7f44a1e0f640 10 monclient(hunting): picked mon.noname-b con 0x7f449c145ae0 addr v1:172.21.15.18:6790/0 2024-03-29T11:15:44.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.206+0000 7f44a1e0f640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f449c145650 con 0x7f449c071de0 2024-03-29T11:15:44.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.206+0000 7f44a1e0f640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f449c0ae260 con 0x7f449c0721c0 2024-03-29T11:15:44.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.206+0000 7f44a1e0f640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f449c1420d0 con 0x7f449c145ae0 2024-03-29T11:15:44.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.206+0000 7f44a1e0f640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:44.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.206+0000 7f44a1e0f640 10 monclient(hunting): authenticate will time out at 1170.478544s 2024-03-29T11:15:44.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.206+0000 7f449b7fe640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f449c071de0 0x7f449c0aed80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:39372/0 (socket says 172.21.15.18:39372) 2024-03-29T11:15:44.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.206+0000 7f449b7fe640 1 -- 172.21.15.18:0/3523424377 learned_addr learned my addr 172.21.15.18:0/3523424377 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:15:44.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.207+0000 7f449a7fc640 1 -- 172.21.15.18:0/3523424377 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1666512450 0 0) 0x7f449c145650 con 0x7f449c071de0 2024-03-29T11:15:44.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.207+0000 7f449a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:15:44.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.207+0000 7f449a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:44.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.207+0000 7f449a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:44.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.207+0000 7f449a7fc640 10 monclient(hunting): my global_id is 4437 2024-03-29T11:15:44.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.207+0000 7f449a7fc640 1 -- 172.21.15.18:0/3523424377 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4484003520 con 0x7f449c071de0 2024-03-29T11:15:44.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.207+0000 7f449a7fc640 1 -- 172.21.15.18:0/3523424377 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2376121142 0 0) 0x7f449c0ae260 con 0x7f449c0721c0 2024-03-29T11:15:44.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.207+0000 7f449a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:15:44.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.207+0000 7f449a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:44.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.207+0000 7f449a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:44.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.207+0000 7f449a7fc640 10 monclient(hunting): my global_id is 4420 2024-03-29T11:15:44.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.207+0000 7f449a7fc640 1 -- 172.21.15.18:0/3523424377 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f449c145650 con 0x7f449c0721c0 2024-03-29T11:15:44.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.207+0000 7f449a7fc640 1 -- 172.21.15.18:0/3523424377 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3873845094 0 0) 0x7f449c1420d0 con 0x7f449c145ae0 2024-03-29T11:15:44.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.207+0000 7f449a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:15:44.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.207+0000 7f449a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:44.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.207+0000 7f449a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:44.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.207+0000 7f449a7fc640 10 monclient(hunting): my global_id is 4505 2024-03-29T11:15:44.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.207+0000 7f449a7fc640 1 -- 172.21.15.18:0/3523424377 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f449c0ae260 con 0x7f449c145ae0 2024-03-29T11:15:44.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.208+0000 7f449a7fc640 1 -- 172.21.15.18:0/3523424377 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3863543880 0 0) 0x7f4484003520 con 0x7f449c071de0 2024-03-29T11:15:44.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.208+0000 7f449a7fc640 1 -- 172.21.15.18:0/3523424377 >> v1:172.21.15.18:6790/0 conn(0x7f449c145ae0 legacy=0x7f449c145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:44.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.208+0000 7f449a7fc640 1 -- 172.21.15.18:0/3523424377 >> v1:172.21.15.70:6789/0 conn(0x7f449c0721c0 legacy=0x7f449c13f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:44.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.208+0000 7f449a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:15:44.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.208+0000 7f449a7fc640 1 monclient: found mon.noname-a 2024-03-29T11:15:44.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.208+0000 7f449a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:44.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.208+0000 7f449a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-29T11:15:44.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.208+0000 7f449a7fc640 1 -- 172.21.15.18:0/3523424377 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f449c0b0590 con 0x7f449c071de0 2024-03-29T11:15:44.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.208+0000 7f449a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:15:44.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.208+0000 7f449a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:15:44.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.208+0000 7f449a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:44.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.208+0000 7f449a7fc640 1 -- 172.21.15.18:0/3523424377 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f448c001690 con 0x7f449c071de0 2024-03-29T11:15:44.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.208+0000 7f449a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:44.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.208+0000 7f449a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:15:44.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.208+0000 7f449a7fc640 10 monclient: dump: 2024-03-29T11:15:44.189 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:15:44.189 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:44.189 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:44.189 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:44.189 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:44.189 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:44.189 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:15:44.189 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:15:44.189 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:15:44.189 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:44.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.208+0000 7f44a1e0f640 5 monclient: authenticate success, global_id 4437 2024-03-29T11:15:44.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.208+0000 7f44a1e0f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:44.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.209+0000 7f449a7fc640 1 -- 172.21.15.18:0/3523424377 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f448c0017f0 con 0x7f449c071de0 2024-03-29T11:15:44.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.209+0000 7f449a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:44.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.209+0000 7f449a7fc640 1 -- 172.21.15.18:0/3523424377 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f448c001b80 con 0x7f449c071de0 2024-03-29T11:15:44.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.209+0000 7f449a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:44.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.209+0000 7f449a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:44.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.209+0000 7f449a7fc640 10 monclient: dump: 2024-03-29T11:15:44.190 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:15:44.190 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:44.190 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:44.190 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:44.190 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:44.190 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:44.190 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:15:44.190 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:15:44.190 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:15:44.190 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:44.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.209+0000 7f44a1e0f640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:44.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.209+0000 7f44a1e0f640 4 set_mon_vals no callback set 2024-03-29T11:15:44.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.210+0000 7f44a1e0f640 10 monclient: shutdown 2024-03-29T11:15:44.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.211+0000 7f44a1e0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:44.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.211+0000 7f44a1e0f640 1 -- 172.21.15.18:0/3523424377 >> v1:172.21.15.18:6789/0 conn(0x7f449c071de0 legacy=0x7f449c0aed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:44.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.211+0000 7f44a1e0f640 1 -- 172.21.15.18:0/3523424377 shutdown_connections 2024-03-29T11:15:44.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.211+0000 7f44a1e0f640 1 -- 172.21.15.18:0/3523424377 >> 172.21.15.18:0/3523424377 conn(0x7f449c06b660 msgr2=0x7f449c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:44.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.211+0000 7f44a1e0f640 1 -- 172.21.15.18:0/3523424377 shutdown_connections 2024-03-29T11:15:44.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.211+0000 7f44a1e0f640 1 -- 172.21.15.18:0/3523424377 wait complete. 2024-03-29T11:15:44.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.212+0000 7f44a1e0f640 10 monclient: build_initial_monmap 2024-03-29T11:15:44.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.212+0000 7f44a1e0f640 10 monclient: monmap: 2024-03-29T11:15:44.201 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:15:44.202 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:44.202 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:15:44.202 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:15:44.202 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:44.202 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:44.202 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:15:44.202 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:15:44.202 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:15:44.202 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:44.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.213+0000 7f44a1e0f640 1 librados: starting msgr at 2024-03-29T11:15:44.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.213+0000 7f44a1e0f640 1 librados: starting objecter 2024-03-29T11:15:44.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.213+0000 7f44a1e0f640 1 Processor -- start 2024-03-29T11:15:44.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.213+0000 7f44a1e0f640 1 -- start start 2024-03-29T11:15:44.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.213+0000 7f44a1e0f640 1 librados: setting wanted keys 2024-03-29T11:15:44.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.213+0000 7f44a1e0f640 1 librados: calling monclient init 2024-03-29T11:15:44.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.213+0000 7f44a1e0f640 10 monclient: init 2024-03-29T11:15:44.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.214+0000 7f44a1e0f640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:44.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.214+0000 7f44a1e0f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:15:44.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.214+0000 7f44a1e0f640 10 monclient(hunting): picked mon.noname-c con 0x7f449c071de0 addr v1:172.21.15.18:6790/0 2024-03-29T11:15:44.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.214+0000 7f44a1e0f640 10 monclient(hunting): picked mon.noname-a con 0x7f449c145ae0 addr v1:172.21.15.18:6789/0 2024-03-29T11:15:44.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.215+0000 7f44a1e0f640 10 monclient(hunting): picked mon.noname-b con 0x7f449c13a330 addr v1:172.21.15.70:6789/0 2024-03-29T11:15:44.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.215+0000 7f44a1e0f640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f449c0ac8d0 con 0x7f449c145ae0 2024-03-29T11:15:44.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.215+0000 7f44a1e0f640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f449c13b110 con 0x7f449c13a330 2024-03-29T11:15:44.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.215+0000 7f44a1e0f640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f449c13b880 con 0x7f449c071de0 2024-03-29T11:15:44.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.215+0000 7f44a1e0f640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:44.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.215+0000 7f44a1e0f640 10 monclient(hunting): authenticate will time out at 1170.487133s 2024-03-29T11:15:44.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.215+0000 7f449bfff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f449c13a330 0x7f449c0aba20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.18:58410/0 (socket says 172.21.15.18:58410) 2024-03-29T11:15:44.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.215+0000 7f449bfff640 1 -- 172.21.15.18:0/2221453110 learned_addr learned my addr 172.21.15.18:0/2221453110 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:15:44.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.215+0000 7f449b7fe640 1 --1- 172.21.15.18:0/2221453110 >> v1:172.21.15.18:6790/0 conn(0x7f449c071de0 0x7f449c13aa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:36106/0 (socket says 172.21.15.18:36106) 2024-03-29T11:15:44.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.215+0000 7f4483fff640 10 client.?.objecter ms_handle_connect 0x7f449c13a330 2024-03-29T11:15:44.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.215+0000 7f4483fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:44.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.215+0000 7f4483fff640 10 client.?.objecter ms_handle_connect 0x7f449c071de0 2024-03-29T11:15:44.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.215+0000 7f4483fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:44.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.215+0000 7f4483fff640 10 client.?.objecter ms_handle_connect 0x7f449c145ae0 2024-03-29T11:15:44.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.215+0000 7f4483fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:44.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.216+0000 7f4483fff640 1 -- 172.21.15.18:0/2221453110 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 20607267 0 0) 0x7f449c13b110 con 0x7f449c13a330 2024-03-29T11:15:44.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.216+0000 7f4483fff640 10 monclient(hunting): _negotiate 2024-03-29T11:15:44.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.216+0000 7f4483fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:44.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.216+0000 7f4483fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:44.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.216+0000 7f4483fff640 10 monclient(hunting): my global_id is 4423 2024-03-29T11:15:44.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.216+0000 7f4483fff640 1 -- 172.21.15.18:0/2221453110 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4488004eb0 con 0x7f449c13a330 2024-03-29T11:15:44.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.216+0000 7f4483fff640 1 -- 172.21.15.18:0/2221453110 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 487627020 0 0) 0x7f449c13b880 con 0x7f449c071de0 2024-03-29T11:15:44.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.216+0000 7f4483fff640 10 monclient(hunting): _negotiate 2024-03-29T11:15:44.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.216+0000 7f4483fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:44.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.216+0000 7f4483fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:44.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.216+0000 7f4483fff640 10 monclient(hunting): my global_id is 4508 2024-03-29T11:15:44.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.216+0000 7f4483fff640 1 -- 172.21.15.18:0/2221453110 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f449c13b110 con 0x7f449c071de0 2024-03-29T11:15:44.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.216+0000 7f4483fff640 1 -- 172.21.15.18:0/2221453110 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2065763063 0 0) 0x7f4488004eb0 con 0x7f449c13a330 2024-03-29T11:15:44.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.216+0000 7f4483fff640 1 -- 172.21.15.18:0/2221453110 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4488005490 con 0x7f449c13a330 2024-03-29T11:15:44.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.216+0000 7f4483fff640 1 -- 172.21.15.18:0/2221453110 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f4494001c80 con 0x7f449c13a330 2024-03-29T11:15:44.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.216+0000 7f4483fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:44.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.216+0000 7f4483fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:15:44.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.216+0000 7f4483fff640 10 monclient(hunting): dump: 2024-03-29T11:15:44.205 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:15:44.205 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:44.205 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:44.205 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:44.205 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:44.206 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:44.206 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:15:44.206 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:15:44.206 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:15:44.206 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:44.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.216+0000 7f4483fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:44.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.216+0000 7f4483fff640 1 -- 172.21.15.18:0/2221453110 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4259108996 0 0) 0x7f449c0ac8d0 con 0x7f449c145ae0 2024-03-29T11:15:44.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.216+0000 7f4483fff640 10 monclient(hunting): _negotiate 2024-03-29T11:15:44.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.216+0000 7f4483fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:44.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.216+0000 7f4483fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:44.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.216+0000 7f4483fff640 10 monclient(hunting): my global_id is 4440 2024-03-29T11:15:44.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.216+0000 7f4483fff640 1 -- 172.21.15.18:0/2221453110 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4488004eb0 con 0x7f449c145ae0 2024-03-29T11:15:44.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.217+0000 7f4483fff640 1 -- 172.21.15.18:0/2221453110 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2962194436 0 0) 0x7f4488004eb0 con 0x7f449c145ae0 2024-03-29T11:15:44.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.217+0000 7f4483fff640 1 -- 172.21.15.18:0/2221453110 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f449c0ac8d0 con 0x7f449c145ae0 2024-03-29T11:15:44.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.217+0000 7f4483fff640 1 -- 172.21.15.18:0/2221453110 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f4490001930 con 0x7f449c145ae0 2024-03-29T11:15:44.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.217+0000 7f4483fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:44.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.217+0000 7f4483fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:44.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.217+0000 7f4483fff640 10 monclient(hunting): dump: 2024-03-29T11:15:44.207 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:15:44.207 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:44.207 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:44.207 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:44.207 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:44.207 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:44.207 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:15:44.207 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:15:44.207 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:15:44.207 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:44.207 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.217+0000 7f4483fff640 1 -- 172.21.15.18:0/2221453110 <== mon.1 v1:172.21.15.70:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2879718460 0 0) 0x7f4488005490 con 0x7f449c13a330 2024-03-29T11:15:44.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.217+0000 7f4483fff640 1 -- 172.21.15.18:0/2221453110 >> v1:172.21.15.18:6790/0 conn(0x7f449c071de0 legacy=0x7f449c13aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:44.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.217+0000 7f4483fff640 1 -- 172.21.15.18:0/2221453110 >> v1:172.21.15.18:6789/0 conn(0x7f449c145ae0 legacy=0x7f449c0ab310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:44.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.217+0000 7f4483fff640 10 monclient: _finish_hunting 0 2024-03-29T11:15:44.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.217+0000 7f4483fff640 1 monclient: found mon.b 2024-03-29T11:15:44.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.217+0000 7f4483fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:44.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.217+0000 7f4483fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:15:44.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.217+0000 7f4483fff640 1 -- 172.21.15.18:0/2221453110 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f449c13bb60 con 0x7f449c13a330 2024-03-29T11:15:44.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.217+0000 7f4483fff640 10 monclient: _finish_auth 0 2024-03-29T11:15:44.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.217+0000 7f4483fff640 10 monclient: _check_auth_tickets 2024-03-29T11:15:44.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.217+0000 7f4483fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:44.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.217+0000 7f4483fff640 1 -- 172.21.15.18:0/2221453110 <== mon.1 v1:172.21.15.70:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4494003970 con 0x7f449c13a330 2024-03-29T11:15:44.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.217+0000 7f4483fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:44.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.217+0000 7f4483fff640 1 -- 172.21.15.18:0/2221453110 <== mon.1 v1:172.21.15.70:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f4494003d40 con 0x7f449c13a330 2024-03-29T11:15:44.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.217+0000 7f4483fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:44.208 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.217+0000 7f4483fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:15:44.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.217+0000 7f4483fff640 10 monclient: dump: 2024-03-29T11:15:44.209 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:15:44.209 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:44.209 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:44.209 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:44.209 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:44.209 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:44.209 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:15:44.209 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:15:44.209 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:15:44.209 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:44.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.217+0000 7f4499ffb640 4 set_mon_vals no callback set 2024-03-29T11:15:44.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.218+0000 7f44a1e0f640 5 monclient: authenticate success, global_id 4423 2024-03-29T11:15:44.209 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.218+0000 7f44a1e0f640 10 monclient: _renew_subs 2024-03-29T11:15:44.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.218+0000 7f44a1e0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:15:44.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.218+0000 7f44a1e0f640 1 -- 172.21.15.18:0/2221453110 --> v1:172.21.15.70:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f449c13be40 con 0x7f449c13a330 2024-03-29T11:15:44.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.218+0000 7f44a1e0f640 10 client.4423.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:44.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.218+0000 7f44a1e0f640 10 monclient: _renew_subs 2024-03-29T11:15:44.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.218+0000 7f44a1e0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:15:44.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.218+0000 7f44a1e0f640 1 -- 172.21.15.18:0/2221453110 --> v1:172.21.15.70:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f449c0ac310 con 0x7f449c13a330 2024-03-29T11:15:44.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.218+0000 7f44a1e0f640 1 librados: init done 2024-03-29T11:15:44.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.220+0000 7f44a1e0f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:44.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.220+0000 7f44a1e0f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:44.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.220+0000 7f44a1e0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:15:44.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.220+0000 7f44a1e0f640 1 -- 172.21.15.18:0/2221453110 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f449c13a7a0 con 0x7f449c13a330 2024-03-29T11:15:44.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.220+0000 7f4483fff640 1 -- 172.21.15.18:0/2221453110 <== mon.1 v1:172.21.15.70:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f4494003ea0 con 0x7f449c13a330 2024-03-29T11:15:44.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.221+0000 7f4483fff640 1 -- 172.21.15.18:0/2221453110 <== mon.1 v1:172.21.15.70:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f4494084dc0 con 0x7f449c13a330 2024-03-29T11:15:44.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.221+0000 7f4483fff640 10 client.4423.objecter ms_dispatch 0x7f449c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:44.210 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.221+0000 7f4483fff640 3 client.4423.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:44.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.221+0000 7f4483fff640 3 client.4423.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:44.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.221+0000 7f4483fff640 20 client.4423.objecter dump_active .. 0 homeless 2024-03-29T11:15:44.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.221+0000 7f449b7fe640 10 monclient: get_auth_request con 0x7f448806e0e0 auth_method 0 2024-03-29T11:15:44.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.222+0000 7f4483fff640 10 client.4423.objecter ms_handle_connect 0x7f448806e0e0 2024-03-29T11:15:44.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.228+0000 7f4483fff640 1 -- 172.21.15.18:0/2221453110 <== mon.1 v1:172.21.15.70:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f44940506a0 con 0x7f449c13a330 2024-03-29T11:15:44.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.228+0000 7f4483fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:44.211 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.228+0000 7f4483fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:44.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.627+0000 7f44a1e0f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:44.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.627+0000 7f44a1e0f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:44.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.627+0000 7f44a1e0f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:15:44.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.627+0000 7f44a1e0f640 1 -- 172.21.15.18:0/2221453110 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f449c075f10 con 0x7f449c13a330 2024-03-29T11:15:44.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.628+0000 7f4483fff640 1 -- 172.21.15.18:0/2221453110 <== mon.1 v1:172.21.15.70:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f44940525d0 con 0x7f449c13a330 2024-03-29T11:15:44.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.628+0000 7f4483fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:44.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.628+0000 7f4483fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:44.606 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:15:44.606 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:44.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.632+0000 7f44a1e0f640 10 librados: watch_flush enter 2024-03-29T11:15:44.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.632+0000 7f44a1e0f640 10 librados: watch_flush exit 2024-03-29T11:15:44.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.632+0000 7f44a1e0f640 20 client.4423.objecter shutdown clearing up homeless session... 2024-03-29T11:15:44.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.632+0000 7f44a1e0f640 10 client.4423.objecter successfully canceled tick 2024-03-29T11:15:44.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.633+0000 7f44a1e0f640 1 -- 172.21.15.18:0/2221453110 >> v1:172.21.15.70:6816/3101871562 conn(0x7f448806e0e0 legacy=0x7f44880705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:44.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.633+0000 7f44a1e0f640 10 monclient: shutdown 2024-03-29T11:15:44.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.633+0000 7f44a1e0f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:44.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.633+0000 7f44a1e0f640 1 -- 172.21.15.18:0/2221453110 >> v1:172.21.15.70:6789/0 conn(0x7f449c13a330 legacy=0x7f449c0aba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:44.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.633+0000 7f44a1e0f640 1 -- 172.21.15.18:0/2221453110 shutdown_connections 2024-03-29T11:15:44.612 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.633+0000 7f44a1e0f640 1 -- 172.21.15.18:0/2221453110 >> 172.21.15.18:0/2221453110 conn(0x7f449c06b660 msgr2=0x7f449c13f0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:44.612 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.633+0000 7f44a1e0f640 1 -- 172.21.15.18:0/2221453110 shutdown_connections 2024-03-29T11:15:44.612 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.633+0000 7f44a1e0f640 1 -- 172.21.15.18:0/2221453110 wait complete. 2024-03-29T11:15:44.612 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:44.633+0000 7f44a1e0f640 1 librados: shutdown 2024-03-29T11:15:44.626 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:51.434 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:51.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.594+0000 7fe79fac0640 10 monclient: get_monmap_and_config 2024-03-29T11:15:51.571 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.594+0000 7fe79fac0640 10 monclient: build_initial_monmap 2024-03-29T11:15:51.572 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.594+0000 7fe79fac0640 10 monclient: monmap: 2024-03-29T11:15:51.572 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:15:51.572 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:51.572 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:15:51.595134+0000 2024-03-29T11:15:51.572 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:15:51.595134+0000 2024-03-29T11:15:51.572 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:51.572 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:51.572 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:15:51.572 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:15:51.572 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:15:51.572 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:51.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.594+0000 7fe79fac0640 1 Processor -- start 2024-03-29T11:15:51.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.595+0000 7fe79fac0640 1 -- start start 2024-03-29T11:15:51.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.595+0000 7fe79fac0640 10 monclient: init 2024-03-29T11:15:51.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.595+0000 7fe79fac0640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:51.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.595+0000 7fe79fac0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:15:51.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.595+0000 7fe79fac0640 10 monclient(hunting): picked mon.noname-a con 0x7fe7981422b0 addr v1:172.21.15.18:6789/0 2024-03-29T11:15:51.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.595+0000 7fe79fac0640 10 monclient(hunting): picked mon.noname-b con 0x7fe798142690 addr v1:172.21.15.18:6790/0 2024-03-29T11:15:51.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.596+0000 7fe79fac0640 10 monclient(hunting): picked mon.noname-c con 0x7fe7980b17d0 addr v1:172.21.15.70:6789/0 2024-03-29T11:15:51.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.596+0000 7fe79fac0640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe798142c80 con 0x7fe7981422b0 2024-03-29T11:15:51.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.596+0000 7fe79fac0640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7980aae40 con 0x7fe7980b17d0 2024-03-29T11:15:51.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.596+0000 7fe79fac0640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe79813fbd0 con 0x7fe798142690 2024-03-29T11:15:51.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.596+0000 7fe79fac0640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:51.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.596+0000 7fe79fac0640 10 monclient(hunting): authenticate will time out at 1177.868031s 2024-03-29T11:15:51.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.596+0000 7fe79d85c640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fe7981422b0 0x7fe79813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:59488/0 (socket says 172.21.15.18:59488) 2024-03-29T11:15:51.574 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.596+0000 7fe79d85c640 1 -- 172.21.15.18:0/1758407851 learned_addr learned my addr 172.21.15.18:0/1758407851 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:15:51.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.597+0000 7fe79c85a640 1 -- 172.21.15.18:0/1758407851 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2525792556 0 0) 0x7fe798142c80 con 0x7fe7981422b0 2024-03-29T11:15:51.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.597+0000 7fe79c85a640 10 monclient(hunting): _negotiate 2024-03-29T11:15:51.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.597+0000 7fe79c85a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:51.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.597+0000 7fe79c85a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:51.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.597+0000 7fe79c85a640 10 monclient(hunting): my global_id is 4443 2024-03-29T11:15:51.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.597+0000 7fe79c85a640 1 -- 172.21.15.18:0/1758407851 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe784003520 con 0x7fe7981422b0 2024-03-29T11:15:51.575 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.597+0000 7fe79c85a640 1 -- 172.21.15.18:0/1758407851 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 475272489 0 0) 0x7fe784003520 con 0x7fe7981422b0 2024-03-29T11:15:51.576 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.597+0000 7fe79c85a640 1 -- 172.21.15.18:0/1758407851 >> v1:172.21.15.18:6790/0 conn(0x7fe798142690 legacy=0x7fe7980b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:15:51.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.599+0000 7fe79c85a640 1 -- 172.21.15.18:0/1758407851 >> v1:172.21.15.70:6789/0 conn(0x7fe7980b17d0 legacy=0x7fe7980b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:51.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.599+0000 7fe79c85a640 10 monclient: _finish_hunting 0 2024-03-29T11:15:51.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.599+0000 7fe79c85a640 1 monclient: found mon.noname-a 2024-03-29T11:15:51.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.599+0000 7fe79c85a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:51.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.599+0000 7fe79c85a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-29T11:15:51.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.599+0000 7fe79c85a640 1 -- 172.21.15.18:0/1758407851 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe798151940 con 0x7fe7981422b0 2024-03-29T11:15:51.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.599+0000 7fe79c85a640 10 monclient: _finish_auth 0 2024-03-29T11:15:51.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.599+0000 7fe79c85a640 10 monclient: _check_auth_tickets 2024-03-29T11:15:51.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.599+0000 7fe79c85a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:51.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.599+0000 7fe79c85a640 1 -- 172.21.15.18:0/1758407851 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fe794001670 con 0x7fe7981422b0 2024-03-29T11:15:51.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.599+0000 7fe79c85a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:51.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.599+0000 7fe79c85a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:15:51.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.599+0000 7fe79c85a640 10 monclient: dump: 2024-03-29T11:15:51.578 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:15:51.578 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:51.578 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:51.578 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:51.578 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:51.578 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:51.578 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:15:51.578 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:15:51.578 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:15:51.578 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:51.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.599+0000 7fe79fac0640 5 monclient: authenticate success, global_id 4443 2024-03-29T11:15:51.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.599+0000 7fe79fac0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:51.578 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.599+0000 7fe79c85a640 1 -- 172.21.15.18:0/1758407851 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe7940017d0 con 0x7fe7981422b0 2024-03-29T11:15:51.579 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.599+0000 7fe79c85a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:51.579 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.599+0000 7fe79fac0640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:51.579 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.599+0000 7fe79fac0640 4 set_mon_vals no callback set 2024-03-29T11:15:51.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.600+0000 7fe79c85a640 1 -- 172.21.15.18:0/1758407851 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fe794001be0 con 0x7fe7981422b0 2024-03-29T11:15:51.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.600+0000 7fe79c85a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:51.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.600+0000 7fe79c85a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:51.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.600+0000 7fe79c85a640 10 monclient: dump: 2024-03-29T11:15:51.580 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:15:51.580 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:51.580 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:51.580 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:51.580 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:51.581 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:51.581 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:15:51.581 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:15:51.581 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:15:51.581 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:51.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.600+0000 7fe79fac0640 10 monclient: shutdown 2024-03-29T11:15:51.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.600+0000 7fe79fac0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:51.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.600+0000 7fe79fac0640 1 -- 172.21.15.18:0/1758407851 >> v1:172.21.15.18:6789/0 conn(0x7fe7981422b0 legacy=0x7fe79813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:51.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.600+0000 7fe79fac0640 1 -- 172.21.15.18:0/1758407851 shutdown_connections 2024-03-29T11:15:51.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.601+0000 7fe79fac0640 1 -- 172.21.15.18:0/1758407851 >> 172.21.15.18:0/1758407851 conn(0x7fe798066a00 msgr2=0x7fe798066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:51.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.601+0000 7fe79fac0640 1 -- 172.21.15.18:0/1758407851 shutdown_connections 2024-03-29T11:15:51.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.601+0000 7fe79fac0640 1 -- 172.21.15.18:0/1758407851 wait complete. 2024-03-29T11:15:51.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.602+0000 7fe79fac0640 10 monclient: build_initial_monmap 2024-03-29T11:15:51.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.602+0000 7fe79fac0640 10 monclient: monmap: 2024-03-29T11:15:51.582 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:15:51.582 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:51.582 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:15:51.582 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:15:51.582 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:51.582 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:51.583 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:15:51.583 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:15:51.583 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:15:51.583 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:51.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.602+0000 7fe79fac0640 1 librados: starting msgr at 2024-03-29T11:15:51.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.602+0000 7fe79fac0640 1 librados: starting objecter 2024-03-29T11:15:51.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.602+0000 7fe79fac0640 1 Processor -- start 2024-03-29T11:15:51.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.602+0000 7fe79fac0640 1 -- start start 2024-03-29T11:15:51.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.602+0000 7fe79fac0640 1 librados: setting wanted keys 2024-03-29T11:15:51.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.602+0000 7fe79fac0640 1 librados: calling monclient init 2024-03-29T11:15:51.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.602+0000 7fe79fac0640 10 monclient: init 2024-03-29T11:15:51.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.603+0000 7fe79fac0640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:51.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.603+0000 7fe79fac0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:15:51.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.603+0000 7fe79fac0640 10 monclient(hunting): picked mon.noname-a con 0x7fe7980b17d0 addr v1:172.21.15.18:6789/0 2024-03-29T11:15:51.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.603+0000 7fe79fac0640 10 monclient(hunting): picked mon.noname-b con 0x7fe7981422b0 addr v1:172.21.15.70:6789/0 2024-03-29T11:15:51.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.603+0000 7fe79fac0640 10 monclient(hunting): picked mon.noname-c con 0x7fe798142690 addr v1:172.21.15.18:6790/0 2024-03-29T11:15:51.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.604+0000 7fe79fac0640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe798132040 con 0x7fe7980b17d0 2024-03-29T11:15:51.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.604+0000 7fe79d85c640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fe7980b17d0 0x7fe7981dba10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:59970/0 (socket says 172.21.15.18:59970) 2024-03-29T11:15:51.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.604+0000 7fe79d85c640 1 -- 172.21.15.18:0/3255356995 learned_addr learned my addr 172.21.15.18:0/3255356995 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:15:51.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.604+0000 7fe79fac0640 1 -- 172.21.15.18:0/3255356995 --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe79806eed0 con 0x7fe7981422b0 2024-03-29T11:15:51.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.604+0000 7fe78e7fc640 10 client.?.objecter ms_handle_connect 0x7fe7980b17d0 2024-03-29T11:15:51.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.604+0000 7fe78e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:51.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.604+0000 7fe79fac0640 1 -- 172.21.15.18:0/3255356995 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe79812f680 con 0x7fe798142690 2024-03-29T11:15:51.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.604+0000 7fe79fac0640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:51.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.604+0000 7fe79fac0640 10 monclient(hunting): authenticate will time out at 1177.876246s 2024-03-29T11:15:51.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.604+0000 7fe78e7fc640 10 client.?.objecter ms_handle_connect 0x7fe7981422b0 2024-03-29T11:15:51.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.604+0000 7fe78e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:51.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.604+0000 7fe78e7fc640 1 -- 172.21.15.18:0/3255356995 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1321695272 0 0) 0x7fe79806eed0 con 0x7fe7981422b0 2024-03-29T11:15:51.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.604+0000 7fe78e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:15:51.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.604+0000 7fe78e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:51.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.604+0000 7fe78e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:51.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.604+0000 7fe78e7fc640 10 monclient(hunting): my global_id is 4429 2024-03-29T11:15:51.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.604+0000 7fe78e7fc640 1 -- 172.21.15.18:0/3255356995 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe77c004eb0 con 0x7fe7981422b0 2024-03-29T11:15:51.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.604+0000 7fe78e7fc640 1 -- 172.21.15.18:0/3255356995 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3576978581 0 0) 0x7fe798132040 con 0x7fe7980b17d0 2024-03-29T11:15:51.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.604+0000 7fe78e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:15:51.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.604+0000 7fe78e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:51.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.604+0000 7fe78e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:51.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.604+0000 7fe78e7fc640 10 monclient(hunting): my global_id is 4446 2024-03-29T11:15:51.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.604+0000 7fe78e7fc640 1 -- 172.21.15.18:0/3255356995 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe79806eed0 con 0x7fe7980b17d0 2024-03-29T11:15:51.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.605+0000 7fe78e7fc640 1 -- 172.21.15.18:0/3255356995 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 11399570 0 0) 0x7fe77c004eb0 con 0x7fe7981422b0 2024-03-29T11:15:51.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.605+0000 7fe78e7fc640 1 -- 172.21.15.18:0/3255356995 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe798132040 con 0x7fe7981422b0 2024-03-29T11:15:51.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.605+0000 7fe78e7fc640 1 -- 172.21.15.18:0/3255356995 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3389246335 0 0) 0x7fe79806eed0 con 0x7fe7980b17d0 2024-03-29T11:15:51.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.605+0000 7fe78e7fc640 1 -- 172.21.15.18:0/3255356995 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe77c004eb0 con 0x7fe7980b17d0 2024-03-29T11:15:51.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.605+0000 7fe78e7fc640 1 -- 172.21.15.18:0/3255356995 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fe7880017d0 con 0x7fe7981422b0 2024-03-29T11:15:51.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.605+0000 7fe78e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:51.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.605+0000 7fe78e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:15:51.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.605+0000 7fe78e7fc640 10 monclient(hunting): dump: 2024-03-29T11:15:51.592 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:15:51.592 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:51.592 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:51.592 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:51.592 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:51.592 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:51.592 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:15:51.592 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:15:51.592 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:15:51.592 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:51.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.605+0000 7fe78e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:51.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.605+0000 7fe78e7fc640 1 -- 172.21.15.18:0/3255356995 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fe794001be0 con 0x7fe7980b17d0 2024-03-29T11:15:51.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.605+0000 7fe78e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:51.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.605+0000 7fe78e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:51.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.605+0000 7fe78e7fc640 10 monclient(hunting): dump: 2024-03-29T11:15:51.593 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:15:51.593 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:51.593 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:51.593 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:51.593 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:51.593 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:51.593 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:15:51.593 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:15:51.593 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:15:51.593 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:51.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.606+0000 7fe78e7fc640 1 -- 172.21.15.18:0/3255356995 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3359515196 0 0) 0x7fe77c004eb0 con 0x7fe7980b17d0 2024-03-29T11:15:51.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.606+0000 7fe78e7fc640 1 -- 172.21.15.18:0/3255356995 >> v1:172.21.15.18:6790/0 conn(0x7fe798142690 legacy=0x7fe79812ef70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:15:51.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.607+0000 7fe78e7fc640 1 -- 172.21.15.18:0/3255356995 >> v1:172.21.15.70:6789/0 conn(0x7fe7981422b0 legacy=0x7fe79812e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:51.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.607+0000 7fe78e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:15:51.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.607+0000 7fe78e7fc640 1 monclient: found mon.a 2024-03-29T11:15:51.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.607+0000 7fe78e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:51.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.607+0000 7fe78e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:15:51.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.607+0000 7fe78e7fc640 1 -- 172.21.15.18:0/3255356995 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7981324d0 con 0x7fe7980b17d0 2024-03-29T11:15:51.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.607+0000 7fe78e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:15:51.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.607+0000 7fe78e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:15:51.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.607+0000 7fe78e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:51.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.607+0000 7fe79fac0640 5 monclient: authenticate success, global_id 4446 2024-03-29T11:15:51.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.607+0000 7fe79fac0640 10 monclient: _renew_subs 2024-03-29T11:15:51.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.607+0000 7fe79fac0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:15:51.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.607+0000 7fe79fac0640 1 -- 172.21.15.18:0/3255356995 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe798131c40 con 0x7fe7980b17d0 2024-03-29T11:15:51.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.607+0000 7fe79fac0640 10 client.4446.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:51.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.607+0000 7fe79fac0640 10 monclient: _renew_subs 2024-03-29T11:15:51.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.607+0000 7fe79fac0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:15:51.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.607+0000 7fe79fac0640 1 -- 172.21.15.18:0/3255356995 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe79806de90 con 0x7fe7980b17d0 2024-03-29T11:15:51.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.607+0000 7fe79fac0640 1 librados: init done 2024-03-29T11:15:51.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.608+0000 7fe78e7fc640 1 -- 172.21.15.18:0/3255356995 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe794004a90 con 0x7fe7980b17d0 2024-03-29T11:15:51.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.608+0000 7fe78e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:51.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.608+0000 7fe78e7fc640 1 -- 172.21.15.18:0/3255356995 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fe794004da0 con 0x7fe7980b17d0 2024-03-29T11:15:51.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.608+0000 7fe78e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:51.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.608+0000 7fe78e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:15:51.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.608+0000 7fe78ffff640 4 set_mon_vals no callback set 2024-03-29T11:15:51.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.608+0000 7fe78e7fc640 10 monclient: dump: 2024-03-29T11:15:51.595 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:15:51.595 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:51.595 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:51.596 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:51.596 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:51.596 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:51.596 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:15:51.596 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:15:51.596 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:15:51.596 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:51.596 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.609+0000 7fe78e7fc640 1 -- 172.21.15.18:0/3255356995 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7fe79401ad70 con 0x7fe7980b17d0 2024-03-29T11:15:51.596 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.609+0000 7fe79fac0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:51.596 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.609+0000 7fe79fac0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:51.596 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.609+0000 7fe79fac0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:15:51.596 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.609+0000 7fe79fac0640 1 -- 172.21.15.18:0/3255356995 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe780005080 con 0x7fe7980b17d0 2024-03-29T11:15:51.596 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.610+0000 7fe78e7fc640 1 -- 172.21.15.18:0/3255356995 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7fe794084b10 con 0x7fe7980b17d0 2024-03-29T11:15:51.596 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.610+0000 7fe78e7fc640 10 client.4446.objecter ms_dispatch 0x7fe798000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:51.596 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.610+0000 7fe78e7fc640 3 client.4446.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:51.597 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.610+0000 7fe78e7fc640 3 client.4446.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:51.597 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.610+0000 7fe78e7fc640 20 client.4446.objecter dump_active .. 0 homeless 2024-03-29T11:15:51.597 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.611+0000 7fe79d05b640 10 monclient: get_auth_request con 0x7fe77c06de30 auth_method 0 2024-03-29T11:15:51.597 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.611+0000 7fe78e7fc640 10 client.4446.objecter ms_handle_connect 0x7fe77c06de30 2024-03-29T11:15:51.597 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.616+0000 7fe78e7fc640 1 -- 172.21.15.18:0/3255356995 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe7940503f0 con 0x7fe7980b17d0 2024-03-29T11:15:51.597 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.616+0000 7fe78e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:51.598 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.616+0000 7fe78e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:51.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.927+0000 7fe79fac0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:51.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.927+0000 7fe79fac0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:51.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.927+0000 7fe79fac0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:15:51.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.927+0000 7fe79fac0640 1 -- 172.21.15.18:0/3255356995 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe780005330 con 0x7fe7980b17d0 2024-03-29T11:15:51.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.928+0000 7fe78e7fc640 1 -- 172.21.15.18:0/3255356995 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7fe794052320 con 0x7fe7980b17d0 2024-03-29T11:15:51.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.928+0000 7fe78e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:51.906 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:15:51.907 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:51.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.928+0000 7fe78e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:51.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.931+0000 7fe79fac0640 10 librados: watch_flush enter 2024-03-29T11:15:51.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.932+0000 7fe79fac0640 10 librados: watch_flush exit 2024-03-29T11:15:51.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.932+0000 7fe79fac0640 20 client.4446.objecter shutdown clearing up homeless session... 2024-03-29T11:15:51.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.932+0000 7fe79fac0640 10 client.4446.objecter successfully canceled tick 2024-03-29T11:15:51.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.932+0000 7fe79fac0640 1 -- 172.21.15.18:0/3255356995 >> v1:172.21.15.70:6816/3101871562 conn(0x7fe77c06de30 legacy=0x7fe77c0702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:51.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.932+0000 7fe79fac0640 10 monclient: shutdown 2024-03-29T11:15:51.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.932+0000 7fe79fac0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:51.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.932+0000 7fe79fac0640 1 -- 172.21.15.18:0/3255356995 >> v1:172.21.15.18:6789/0 conn(0x7fe7980b17d0 legacy=0x7fe7981dba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:51.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.932+0000 7fe79fac0640 1 -- 172.21.15.18:0/3255356995 shutdown_connections 2024-03-29T11:15:51.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.932+0000 7fe79fac0640 1 -- 172.21.15.18:0/3255356995 >> 172.21.15.18:0/3255356995 conn(0x7fe798066a00 msgr2=0x7fe79813b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:51.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.932+0000 7fe79fac0640 1 -- 172.21.15.18:0/3255356995 shutdown_connections 2024-03-29T11:15:51.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.932+0000 7fe79fac0640 1 -- 172.21.15.18:0/3255356995 wait complete. 2024-03-29T11:15:51.911 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:51.932+0000 7fe79fac0640 1 librados: shutdown 2024-03-29T11:15:51.923 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:15:58.730 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:15:58.877 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.899+0000 7fedc1b1c640 10 monclient: get_monmap_and_config 2024-03-29T11:15:58.877 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.899+0000 7fedc1b1c640 10 monclient: build_initial_monmap 2024-03-29T11:15:58.878 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.900+0000 7fedc1b1c640 10 monclient: monmap: 2024-03-29T11:15:58.878 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:15:58.878 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:58.878 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:15:58.900927+0000 2024-03-29T11:15:58.878 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:15:58.900927+0000 2024-03-29T11:15:58.878 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:58.878 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:58.878 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:15:58.878 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:15:58.878 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:15:58.878 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:58.881 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.901+0000 7fedc1b1c640 1 Processor -- start 2024-03-29T11:15:58.881 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.901+0000 7fedc1b1c640 1 -- start start 2024-03-29T11:15:58.881 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.901+0000 7fedc1b1c640 10 monclient: init 2024-03-29T11:15:58.881 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.902+0000 7fedc1b1c640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:58.883 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.902+0000 7fedc1b1c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:15:58.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.902+0000 7fedc1b1c640 10 monclient(hunting): picked mon.noname-a con 0x7fedbc1422b0 addr v1:172.21.15.18:6789/0 2024-03-29T11:15:58.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.902+0000 7fedc1b1c640 10 monclient(hunting): picked mon.noname-b con 0x7fedbc142690 addr v1:172.21.15.18:6790/0 2024-03-29T11:15:58.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.902+0000 7fedc1b1c640 10 monclient(hunting): picked mon.noname-c con 0x7fedbc0b17d0 addr v1:172.21.15.70:6789/0 2024-03-29T11:15:58.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.902+0000 7fedc1b1c640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedbc142c80 con 0x7fedbc1422b0 2024-03-29T11:15:58.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.902+0000 7fedc1b1c640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedbc0aae40 con 0x7fedbc0b17d0 2024-03-29T11:15:58.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.902+0000 7fedc1b1c640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedbc13fbd0 con 0x7fedbc142690 2024-03-29T11:15:58.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.902+0000 7fedc1b1c640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:58.884 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.902+0000 7fedc1b1c640 10 monclient(hunting): authenticate will time out at 1185.174808s 2024-03-29T11:15:58.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.907+0000 7fedbb7fe640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fedbc1422b0 0x7fedbc13f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:43347/0 (socket says 172.21.15.18:43347) 2024-03-29T11:15:58.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.907+0000 7fedbb7fe640 1 -- 172.21.15.18:0/3565984597 learned_addr learned my addr 172.21.15.18:0/3565984597 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:15:58.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.908+0000 7fedba7fc640 1 -- 172.21.15.18:0/3565984597 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 268441184 0 0) 0x7fedbc142c80 con 0x7fedbc1422b0 2024-03-29T11:15:58.885 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.908+0000 7fedba7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:15:58.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.908+0000 7fedba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:58.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.908+0000 7fedba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:58.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.908+0000 7fedba7fc640 10 monclient(hunting): my global_id is 4449 2024-03-29T11:15:58.886 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.908+0000 7fedba7fc640 1 -- 172.21.15.18:0/3565984597 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feda4003520 con 0x7fedbc1422b0 2024-03-29T11:15:58.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.911+0000 7fedba7fc640 1 -- 172.21.15.18:0/3565984597 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3025700633 0 0) 0x7feda4003520 con 0x7fedbc1422b0 2024-03-29T11:15:58.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.911+0000 7fedba7fc640 1 -- 172.21.15.18:0/3565984597 >> v1:172.21.15.18:6790/0 conn(0x7fedbc142690 legacy=0x7fedbc0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:58.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.911+0000 7fedba7fc640 1 -- 172.21.15.18:0/3565984597 >> v1:172.21.15.70:6789/0 conn(0x7fedbc0b17d0 legacy=0x7fedbc0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:15:58.889 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.911+0000 7fedba7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:15:58.890 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.911+0000 7fedba7fc640 1 monclient: found mon.noname-a 2024-03-29T11:15:58.890 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.911+0000 7fedba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:58.890 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.911+0000 7fedba7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-29T11:15:58.890 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.911+0000 7fedba7fc640 1 -- 172.21.15.18:0/3565984597 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedbc151940 con 0x7fedbc1422b0 2024-03-29T11:15:58.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.911+0000 7fedba7fc640 10 monclient: _finish_auth 0 2024-03-29T11:15:58.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.911+0000 7fedba7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:15:58.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.911+0000 7fedba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:58.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.911+0000 7fedc1b1c640 5 monclient: authenticate success, global_id 4449 2024-03-29T11:15:58.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.911+0000 7fedc1b1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:58.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.911+0000 7fedba7fc640 1 -- 172.21.15.18:0/3565984597 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fedac001690 con 0x7fedbc1422b0 2024-03-29T11:15:58.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.911+0000 7fedba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:58.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.912+0000 7fedba7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:15:58.891 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.912+0000 7fedba7fc640 10 monclient: dump: 2024-03-29T11:15:58.891 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:15:58.891 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:58.891 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:58.891 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:58.891 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:58.892 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:58.892 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:15:58.892 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:15:58.892 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:15:58.892 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:58.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.912+0000 7fedc1b1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:15:58.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.912+0000 7fedba7fc640 1 -- 172.21.15.18:0/3565984597 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fedac0018b0 con 0x7fedbc1422b0 2024-03-29T11:15:58.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.912+0000 7fedba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:58.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.912+0000 7fedc1b1c640 10 monclient: get_monmap_and_config success 2024-03-29T11:15:58.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.912+0000 7fedc1b1c640 4 set_mon_vals no callback set 2024-03-29T11:15:58.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.913+0000 7fedc1b1c640 10 monclient: shutdown 2024-03-29T11:15:58.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.913+0000 7fedc1b1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:58.892 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.913+0000 7fedc1b1c640 1 -- 172.21.15.18:0/3565984597 >> v1:172.21.15.18:6789/0 conn(0x7fedbc1422b0 legacy=0x7fedbc13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:58.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.915+0000 7fedc1b1c640 1 -- 172.21.15.18:0/3565984597 shutdown_connections 2024-03-29T11:15:58.893 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.915+0000 7fedc1b1c640 1 -- 172.21.15.18:0/3565984597 >> 172.21.15.18:0/3565984597 conn(0x7fedbc066a00 msgr2=0x7fedbc066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:58.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.916+0000 7fedc1b1c640 1 -- 172.21.15.18:0/3565984597 shutdown_connections 2024-03-29T11:15:58.894 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.917+0000 7fedc1b1c640 1 -- 172.21.15.18:0/3565984597 wait complete. 2024-03-29T11:15:58.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.918+0000 7fedc1b1c640 10 monclient: build_initial_monmap 2024-03-29T11:15:58.896 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.918+0000 7fedc1b1c640 10 monclient: monmap: 2024-03-29T11:15:58.896 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:15:58.896 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:15:58.896 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:15:58.896 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:15:58.896 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:15:58.896 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:58.896 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:15:58.896 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:15:58.896 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:15:58.896 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:58.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.918+0000 7fedc1b1c640 1 librados: starting msgr at 2024-03-29T11:15:58.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.918+0000 7fedc1b1c640 1 librados: starting objecter 2024-03-29T11:15:58.897 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.918+0000 7fedc1b1c640 1 Processor -- start 2024-03-29T11:15:58.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.919+0000 7fedc1b1c640 1 -- start start 2024-03-29T11:15:58.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.919+0000 7fedc1b1c640 1 librados: setting wanted keys 2024-03-29T11:15:58.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.919+0000 7fedc1b1c640 1 librados: calling monclient init 2024-03-29T11:15:58.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.919+0000 7fedc1b1c640 10 monclient: init 2024-03-29T11:15:58.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.919+0000 7fedc1b1c640 10 monclient: _reopen_session rank -1 2024-03-29T11:15:58.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.920+0000 7fedc1b1c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:15:58.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.920+0000 7fedc1b1c640 10 monclient(hunting): picked mon.noname-c con 0x7fedbc0b17d0 addr v1:172.21.15.18:6790/0 2024-03-29T11:15:58.898 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.920+0000 7fedc1b1c640 10 monclient(hunting): picked mon.noname-a con 0x7fedbc1422b0 addr v1:172.21.15.18:6789/0 2024-03-29T11:15:58.900 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.923+0000 7fedc1b1c640 10 monclient(hunting): picked mon.noname-b con 0x7fedbc142690 addr v1:172.21.15.70:6789/0 2024-03-29T11:15:58.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.923+0000 7fedc1b1c640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedbc12f680 con 0x7fedbc1422b0 2024-03-29T11:15:58.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.923+0000 7fedc1b1c640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedbc06e910 con 0x7fedbc142690 2024-03-29T11:15:58.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.923+0000 7fedc1b1c640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedbc06ebf0 con 0x7fedbc0b17d0 2024-03-29T11:15:58.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.923+0000 7fedc1b1c640 10 monclient(hunting): _renew_subs 2024-03-29T11:15:58.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.923+0000 7fedc1b1c640 10 monclient(hunting): authenticate will time out at 1185.195030s 2024-03-29T11:15:58.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.925+0000 7fedbaffd640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fedbc1422b0 0x7fedbc12e860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:43537/0 (socket says 172.21.15.18:43537) 2024-03-29T11:15:58.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.925+0000 7fedbaffd640 1 -- 172.21.15.18:0/197342482 learned_addr learned my addr 172.21.15.18:0/197342482 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:15:58.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.925+0000 7feda3fff640 10 client.?.objecter ms_handle_connect 0x7fedbc1422b0 2024-03-29T11:15:58.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.925+0000 7feda3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:58.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.926+0000 7feda3fff640 10 client.?.objecter ms_handle_connect 0x7fedbc142690 2024-03-29T11:15:58.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.926+0000 7feda3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:15:58.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.927+0000 7feda3fff640 1 -- 172.21.15.18:0/197342482 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 59551857 0 0) 0x7fedbc06e910 con 0x7fedbc142690 2024-03-29T11:15:58.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.927+0000 7feda3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:15:58.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.927+0000 7feda3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:15:58.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.927+0000 7feda3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:15:58.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.927+0000 7feda3fff640 10 monclient(hunting): my global_id is 4432 2024-03-29T11:15:58.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.927+0000 7feda3fff640 1 -- 172.21.15.18:0/197342482 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feda8004eb0 con 0x7fedbc142690 2024-03-29T11:15:58.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.927+0000 7feda3fff640 1 -- 172.21.15.18:0/197342482 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1476707556 0 0) 0x7feda8004eb0 con 0x7fedbc142690 2024-03-29T11:15:58.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.927+0000 7feda3fff640 1 -- 172.21.15.18:0/197342482 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fedbc06e910 con 0x7fedbc142690 2024-03-29T11:15:58.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.927+0000 7feda3fff640 1 -- 172.21.15.18:0/197342482 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fedb0001510 con 0x7fedbc142690 2024-03-29T11:15:58.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.927+0000 7feda3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:58.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.927+0000 7feda3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:15:58.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.927+0000 7feda3fff640 10 monclient(hunting): dump: 2024-03-29T11:15:58.906 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:15:58.906 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:58.906 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:58.907 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:58.907 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:58.907 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:58.907 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:15:58.907 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:15:58.907 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:15:58.907 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:58.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.927+0000 7feda3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:15:58.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.927+0000 7feda3fff640 1 -- 172.21.15.18:0/197342482 <== mon.1 v1:172.21.15.70:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3625092896 0 0) 0x7fedbc06e910 con 0x7fedbc142690 2024-03-29T11:15:58.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.927+0000 7feda3fff640 1 -- 172.21.15.18:0/197342482 >> v1:172.21.15.18:6790/0 conn(0x7fedbc0b17d0 legacy=0x7fedbc13d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:58.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.927+0000 7feda3fff640 1 -- 172.21.15.18:0/197342482 >> v1:172.21.15.18:6789/0 conn(0x7fedbc1422b0 legacy=0x7fedbc12e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:58.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.928+0000 7feda3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:15:58.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.928+0000 7feda3fff640 1 monclient: found mon.b 2024-03-29T11:15:58.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.928+0000 7feda3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:15:58.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.928+0000 7feda3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:15:58.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.928+0000 7feda3fff640 1 -- 172.21.15.18:0/197342482 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedbc06eed0 con 0x7fedbc142690 2024-03-29T11:15:58.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.928+0000 7feda3fff640 10 monclient: _finish_auth 0 2024-03-29T11:15:58.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.928+0000 7feda3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:15:58.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.928+0000 7feda3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:15:58.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.928+0000 7fedc1b1c640 5 monclient: authenticate success, global_id 4432 2024-03-29T11:15:58.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.928+0000 7fedc1b1c640 10 monclient: _renew_subs 2024-03-29T11:15:58.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.928+0000 7fedc1b1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:15:58.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.928+0000 7fedc1b1c640 1 -- 172.21.15.18:0/197342482 --> v1:172.21.15.70:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedbc0b6a90 con 0x7fedbc142690 2024-03-29T11:15:58.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.928+0000 7fedc1b1c640 10 client.4432.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:15:58.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.928+0000 7fedc1b1c640 10 monclient: _renew_subs 2024-03-29T11:15:58.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.928+0000 7fedc1b1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:15:58.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.928+0000 7fedc1b1c640 1 -- 172.21.15.18:0/197342482 --> v1:172.21.15.70:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fedbc0b6570 con 0x7fedbc142690 2024-03-29T11:15:58.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.928+0000 7fedc1b1c640 1 librados: init done 2024-03-29T11:15:58.909 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.928+0000 7feda3fff640 1 -- 172.21.15.18:0/197342482 <== mon.1 v1:172.21.15.70:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fedb00017a0 con 0x7fedbc142690 2024-03-29T11:15:58.909 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.928+0000 7feda3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:15:58.909 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.928+0000 7feda3fff640 1 -- 172.21.15.18:0/197342482 <== mon.1 v1:172.21.15.70:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fedb0003890 con 0x7fedbc142690 2024-03-29T11:15:58.909 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.928+0000 7feda3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:15:58.909 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.928+0000 7feda3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:15:58.909 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.928+0000 7feda3fff640 10 monclient: dump: 2024-03-29T11:15:58.909 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:15:58.909 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:15:58.909 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:58.910 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:15:58.910 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:15:58.910 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:15:58.910 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:15:58.910 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:15:58.910 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:15:58.910 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:15:58.910 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.929+0000 7fedb9ffb640 4 set_mon_vals no callback set 2024-03-29T11:15:58.914 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.931+0000 7feda3fff640 1 -- 172.21.15.18:0/197342482 <== mon.1 v1:172.21.15.70:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7fedb0019860 con 0x7fedbc142690 2024-03-29T11:15:58.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.935+0000 7feda3fff640 1 -- 172.21.15.18:0/197342482 <== mon.1 v1:172.21.15.70:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7fedb00536a0 con 0x7fedbc142690 2024-03-29T11:15:58.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.935+0000 7feda3fff640 10 client.4432.objecter ms_dispatch 0x7fedbc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:15:58.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.935+0000 7feda3fff640 3 client.4432.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:15:58.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.935+0000 7feda3fff640 3 client.4432.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:15:58.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.935+0000 7feda3fff640 20 client.4432.objecter dump_active .. 0 homeless 2024-03-29T11:15:58.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.935+0000 7fedc1b1c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:58.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.935+0000 7fedc1b1c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:58.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.935+0000 7fedc1b1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:15:58.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.936+0000 7fedc1b1c640 1 -- 172.21.15.18:0/197342482 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed9c005080 con 0x7fedbc142690 2024-03-29T11:15:58.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.936+0000 7fedbb7fe640 10 monclient: get_auth_request con 0x7feda806dc60 auth_method 0 2024-03-29T11:15:58.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.936+0000 7feda3fff640 10 client.4432.objecter ms_handle_connect 0x7feda806dc60 2024-03-29T11:15:58.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.945+0000 7feda3fff640 1 -- 172.21.15.18:0/197342482 <== mon.1 v1:172.21.15.70:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fedb004f940 con 0x7fedbc142690 2024-03-29T11:15:58.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.945+0000 7feda3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:15:58.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:58.945+0000 7feda3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:15:59.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:59.274+0000 7fedc1b1c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:59.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:59.274+0000 7fedc1b1c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:59.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:59.274+0000 7fedc1b1c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:15:59.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:59.274+0000 7fedc1b1c640 1 -- 172.21.15.18:0/197342482 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed9c005330 con 0x7fedbc142690 2024-03-29T11:15:59.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:59.275+0000 7feda3fff640 1 -- 172.21.15.18:0/197342482 <== mon.1 v1:172.21.15.70:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7fedb004fae0 con 0x7fedbc142690 2024-03-29T11:15:59.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:59.275+0000 7feda3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:15:59.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:59.275+0000 7feda3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:15:59.253 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:15:59.253 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:15:59.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:59.278+0000 7fedc1b1c640 10 librados: watch_flush enter 2024-03-29T11:15:59.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:59.278+0000 7fedc1b1c640 10 librados: watch_flush exit 2024-03-29T11:15:59.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:59.278+0000 7fedc1b1c640 20 client.4432.objecter shutdown clearing up homeless session... 2024-03-29T11:15:59.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:59.278+0000 7fedc1b1c640 10 client.4432.objecter successfully canceled tick 2024-03-29T11:15:59.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:59.279+0000 7fedc1b1c640 1 -- 172.21.15.18:0/197342482 >> v1:172.21.15.70:6816/3101871562 conn(0x7feda806dc60 legacy=0x7feda8070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:59.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:59.279+0000 7fedc1b1c640 10 monclient: shutdown 2024-03-29T11:15:59.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:59.279+0000 7fedc1b1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:15:59.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:59.279+0000 7fedc1b1c640 1 -- 172.21.15.18:0/197342482 >> v1:172.21.15.70:6789/0 conn(0x7fedbc142690 legacy=0x7fedbc12ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:15:59.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:59.280+0000 7fedc1b1c640 1 -- 172.21.15.18:0/197342482 shutdown_connections 2024-03-29T11:15:59.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:59.280+0000 7fedc1b1c640 1 -- 172.21.15.18:0/197342482 >> 172.21.15.18:0/197342482 conn(0x7fedbc066a00 msgr2=0x7fedbc0b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:15:59.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:59.280+0000 7fedc1b1c640 1 -- 172.21.15.18:0/197342482 shutdown_connections 2024-03-29T11:15:59.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:59.280+0000 7fedc1b1c640 1 -- 172.21.15.18:0/197342482 wait complete. 2024-03-29T11:15:59.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:15:59.280+0000 7fedc1b1c640 1 librados: shutdown 2024-03-29T11:15:59.278 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:06.085 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:06.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.257+0000 7f17e3732640 10 monclient: get_monmap_and_config 2024-03-29T11:16:06.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.257+0000 7f17e3732640 10 monclient: build_initial_monmap 2024-03-29T11:16:06.235 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.257+0000 7f17e3732640 10 monclient: monmap: 2024-03-29T11:16:06.235 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:16:06.235 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:06.235 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:16:06.258322+0000 2024-03-29T11:16:06.235 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:16:06.258322+0000 2024-03-29T11:16:06.235 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:06.235 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:06.235 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:16:06.235 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:16:06.235 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:16:06.235 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:06.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.259+0000 7f17e3732640 1 Processor -- start 2024-03-29T11:16:06.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.259+0000 7f17e3732640 1 -- start start 2024-03-29T11:16:06.236 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.259+0000 7f17e3732640 10 monclient: init 2024-03-29T11:16:06.238 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.260+0000 7f17e3732640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:06.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.260+0000 7f17e3732640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:16:06.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.260+0000 7f17e3732640 10 monclient(hunting): picked mon.noname-a con 0x7f17dc1457f0 addr v1:172.21.15.18:6789/0 2024-03-29T11:16:06.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.260+0000 7f17e3732640 10 monclient(hunting): picked mon.noname-b con 0x7f17dc13f650 addr v1:172.21.15.18:6790/0 2024-03-29T11:16:06.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.260+0000 7f17e3732640 10 monclient(hunting): picked mon.noname-c con 0x7f17dc0a9d70 addr v1:172.21.15.70:6789/0 2024-03-29T11:16:06.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.260+0000 7f17e3732640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17dc0aaef0 con 0x7f17dc1457f0 2024-03-29T11:16:06.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.260+0000 7f17e3732640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17dc145bd0 con 0x7f17dc0a9d70 2024-03-29T11:16:06.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.260+0000 7f17e3732640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17dc145eb0 con 0x7f17dc13f650 2024-03-29T11:16:06.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.260+0000 7f17e3732640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:06.239 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.260+0000 7f17e3732640 10 monclient(hunting): authenticate will time out at 1192.532826s 2024-03-29T11:16:06.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.262+0000 7f17e0ccd640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f17dc13f650 0x7f17dc0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:33710/0 (socket says 172.21.15.18:33710) 2024-03-29T11:16:06.240 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.262+0000 7f17e0ccd640 1 -- 172.21.15.18:0/3542694761 learned_addr learned my addr 172.21.15.18:0/3542694761 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:16:06.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.263+0000 7f17d3fff640 1 -- 172.21.15.18:0/3542694761 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2274700990 0 0) 0x7f17dc0aaef0 con 0x7f17dc1457f0 2024-03-29T11:16:06.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.263+0000 7f17d3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:16:06.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.263+0000 7f17d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:06.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.263+0000 7f17d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:06.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.263+0000 7f17d3fff640 10 monclient(hunting): my global_id is 4455 2024-03-29T11:16:06.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.263+0000 7f17d3fff640 1 -- 172.21.15.18:0/3542694761 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17c8003520 con 0x7f17dc1457f0 2024-03-29T11:16:06.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.263+0000 7f17d3fff640 1 -- 172.21.15.18:0/3542694761 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2932960837 0 0) 0x7f17dc145eb0 con 0x7f17dc13f650 2024-03-29T11:16:06.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.263+0000 7f17d3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:16:06.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.263+0000 7f17d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:06.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.263+0000 7f17d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:06.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.263+0000 7f17d3fff640 10 monclient(hunting): my global_id is 4514 2024-03-29T11:16:06.244 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.263+0000 7f17d3fff640 1 -- 172.21.15.18:0/3542694761 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17c80048a0 con 0x7f17dc13f650 2024-03-29T11:16:06.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.264+0000 7f17d3fff640 1 -- 172.21.15.18:0/3542694761 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 926607361 0 0) 0x7f17c80048a0 con 0x7f17dc13f650 2024-03-29T11:16:06.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.264+0000 7f17d3fff640 1 -- 172.21.15.18:0/3542694761 >> v1:172.21.15.70:6789/0 conn(0x7f17dc0a9d70 legacy=0x7f17dc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:16:06.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.264+0000 7f17d3fff640 1 -- 172.21.15.18:0/3542694761 >> v1:172.21.15.18:6789/0 conn(0x7f17dc1457f0 legacy=0x7f17dc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:06.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.264+0000 7f17d3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:16:06.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.264+0000 7f17d3fff640 1 monclient: found mon.noname-b 2024-03-29T11:16:06.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.264+0000 7f17d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:06.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.264+0000 7f17d3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:16:06.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.264+0000 7f17d3fff640 1 -- 172.21.15.18:0/3542694761 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17dc146340 con 0x7f17dc13f650 2024-03-29T11:16:06.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.264+0000 7f17d3fff640 10 monclient: _finish_auth 0 2024-03-29T11:16:06.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.264+0000 7f17d3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:16:06.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.264+0000 7f17d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:06.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.264+0000 7f17d3fff640 1 -- 172.21.15.18:0/3542694761 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f17d8001630 con 0x7f17dc13f650 2024-03-29T11:16:06.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.264+0000 7f17d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:06.245 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.264+0000 7f17d3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:16:06.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.264+0000 7f17d3fff640 10 monclient: dump: 2024-03-29T11:16:06.246 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:16:06.246 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:06.246 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:06.246 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:06.246 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:06.246 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:06.246 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:16:06.246 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:16:06.246 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:16:06.246 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:06.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.264+0000 7f17d3fff640 1 -- 172.21.15.18:0/3542694761 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17d8001880 con 0x7f17dc13f650 2024-03-29T11:16:06.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.264+0000 7f17d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:06.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.264+0000 7f17d3fff640 1 -- 172.21.15.18:0/3542694761 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f17d8001630 con 0x7f17dc13f650 2024-03-29T11:16:06.246 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.264+0000 7f17d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:06.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.264+0000 7f17d3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:06.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.264+0000 7f17d3fff640 10 monclient: dump: 2024-03-29T11:16:06.247 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:16:06.247 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:06.247 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:06.247 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:06.247 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:06.247 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:06.247 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:16:06.247 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:16:06.247 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:16:06.247 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:06.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.264+0000 7f17e3732640 5 monclient: authenticate success, global_id 4514 2024-03-29T11:16:06.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.264+0000 7f17e3732640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:06.247 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.264+0000 7f17e3732640 4 set_mon_vals no callback set 2024-03-29T11:16:06.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.265+0000 7f17e3732640 10 monclient: shutdown 2024-03-29T11:16:06.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.265+0000 7f17e3732640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:06.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.265+0000 7f17e3732640 1 -- 172.21.15.18:0/3542694761 >> v1:172.21.15.18:6790/0 conn(0x7f17dc13f650 legacy=0x7f17dc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:06.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.266+0000 7f17e3732640 1 -- 172.21.15.18:0/3542694761 shutdown_connections 2024-03-29T11:16:06.249 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.266+0000 7f17e3732640 1 -- 172.21.15.18:0/3542694761 >> 172.21.15.18:0/3542694761 conn(0x7f17dc066a00 msgr2=0x7f17dc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:06.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.268+0000 7f17e3732640 1 -- 172.21.15.18:0/3542694761 shutdown_connections 2024-03-29T11:16:06.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.269+0000 7f17e3732640 1 -- 172.21.15.18:0/3542694761 wait complete. 2024-03-29T11:16:06.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.270+0000 7f17e3732640 10 monclient: build_initial_monmap 2024-03-29T11:16:06.250 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.270+0000 7f17e3732640 10 monclient: monmap: 2024-03-29T11:16:06.250 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:16:06.250 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:06.250 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:16:06.250 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:16:06.250 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:06.250 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:06.250 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:16:06.250 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:16:06.250 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:16:06.250 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:06.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.270+0000 7f17e3732640 1 librados: starting msgr at 2024-03-29T11:16:06.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.270+0000 7f17e3732640 1 librados: starting objecter 2024-03-29T11:16:06.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.270+0000 7f17e3732640 1 Processor -- start 2024-03-29T11:16:06.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.271+0000 7f17e3732640 1 -- start start 2024-03-29T11:16:06.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.271+0000 7f17e3732640 1 librados: setting wanted keys 2024-03-29T11:16:06.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.271+0000 7f17e3732640 1 librados: calling monclient init 2024-03-29T11:16:06.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.271+0000 7f17e3732640 10 monclient: init 2024-03-29T11:16:06.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.272+0000 7f17e3732640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:06.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.272+0000 7f17e3732640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:16:06.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.272+0000 7f17e3732640 10 monclient(hunting): picked mon.noname-b con 0x7f17dc0a9d70 addr v1:172.21.15.70:6789/0 2024-03-29T11:16:06.251 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.272+0000 7f17e3732640 10 monclient(hunting): picked mon.noname-c con 0x7f17dc13f650 addr v1:172.21.15.18:6790/0 2024-03-29T11:16:06.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.275+0000 7f17e3732640 10 monclient(hunting): picked mon.noname-a con 0x7f17dc1457f0 addr v1:172.21.15.18:6789/0 2024-03-29T11:16:06.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.275+0000 7f17e1ccf640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f17dc1457f0 0x7f17dc1e2ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:54347/0 (socket says 172.21.15.18:54347) 2024-03-29T11:16:06.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.275+0000 7f17e1ccf640 1 -- 172.21.15.18:0/4192093136 learned_addr learned my addr 172.21.15.18:0/4192093136 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:16:06.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.275+0000 7f17e3732640 1 -- 172.21.15.18:0/4192093136 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17dc138cc0 con 0x7f17dc1457f0 2024-03-29T11:16:06.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.275+0000 7f17d1ffb640 10 client.?.objecter ms_handle_connect 0x7f17dc1457f0 2024-03-29T11:16:06.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.275+0000 7f17d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:06.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.275+0000 7f17e3732640 1 -- 172.21.15.18:0/4192093136 --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17dc0ad170 con 0x7f17dc0a9d70 2024-03-29T11:16:06.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.276+0000 7f17e3732640 1 -- 172.21.15.18:0/4192093136 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17dc0ad450 con 0x7f17dc13f650 2024-03-29T11:16:06.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.276+0000 7f17e3732640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:06.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.276+0000 7f17e3732640 10 monclient(hunting): authenticate will time out at 1192.548204s 2024-03-29T11:16:06.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.276+0000 7f17d1ffb640 10 client.?.objecter ms_handle_connect 0x7f17dc13f650 2024-03-29T11:16:06.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.276+0000 7f17d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:06.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.276+0000 7f17d1ffb640 10 client.?.objecter ms_handle_connect 0x7f17dc0a9d70 2024-03-29T11:16:06.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.276+0000 7f17d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:06.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.276+0000 7f17d1ffb640 1 -- 172.21.15.18:0/4192093136 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 972234296 0 0) 0x7f17dc138cc0 con 0x7f17dc1457f0 2024-03-29T11:16:06.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.276+0000 7f17d1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:16:06.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.276+0000 7f17d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:06.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.276+0000 7f17d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:06.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.276+0000 7f17d1ffb640 10 monclient(hunting): my global_id is 4458 2024-03-29T11:16:06.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.276+0000 7f17d1ffb640 1 -- 172.21.15.18:0/4192093136 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17c0004eb0 con 0x7f17dc1457f0 2024-03-29T11:16:06.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.277+0000 7f17d1ffb640 1 -- 172.21.15.18:0/4192093136 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 365968476 0 0) 0x7f17dc0ad450 con 0x7f17dc13f650 2024-03-29T11:16:06.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.277+0000 7f17d1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:16:06.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.277+0000 7f17d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:06.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.277+0000 7f17d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:06.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.277+0000 7f17d1ffb640 10 monclient(hunting): my global_id is 4517 2024-03-29T11:16:06.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.277+0000 7f17d1ffb640 1 -- 172.21.15.18:0/4192093136 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17dc138cc0 con 0x7f17dc13f650 2024-03-29T11:16:06.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.277+0000 7f17d1ffb640 1 -- 172.21.15.18:0/4192093136 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2937537379 0 0) 0x7f17c0004eb0 con 0x7f17dc1457f0 2024-03-29T11:16:06.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.277+0000 7f17d1ffb640 1 -- 172.21.15.18:0/4192093136 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17dc0ad450 con 0x7f17dc1457f0 2024-03-29T11:16:06.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.277+0000 7f17d1ffb640 1 -- 172.21.15.18:0/4192093136 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 503785167 0 0) 0x7f17dc0ad170 con 0x7f17dc0a9d70 2024-03-29T11:16:06.256 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.277+0000 7f17d1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:16:06.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.277+0000 7f17d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:06.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.277+0000 7f17d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:06.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.277+0000 7f17d1ffb640 10 monclient(hunting): my global_id is 4435 2024-03-29T11:16:06.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.277+0000 7f17d1ffb640 1 -- 172.21.15.18:0/4192093136 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17c0004eb0 con 0x7f17dc0a9d70 2024-03-29T11:16:06.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.277+0000 7f17d1ffb640 1 -- 172.21.15.18:0/4192093136 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f17d4001be0 con 0x7f17dc1457f0 2024-03-29T11:16:06.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.277+0000 7f17d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:06.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.277+0000 7f17d1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:16:06.257 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.277+0000 7f17d1ffb640 10 monclient(hunting): dump: 2024-03-29T11:16:06.257 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:16:06.257 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:06.257 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:06.257 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:06.257 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:06.257 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:06.257 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:16:06.257 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:16:06.258 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:16:06.258 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:06.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.277+0000 7f17d1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:06.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.278+0000 7f17d1ffb640 1 -- 172.21.15.18:0/4192093136 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 308525401 0 0) 0x7f17dc138cc0 con 0x7f17dc13f650 2024-03-29T11:16:06.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.278+0000 7f17d1ffb640 1 -- 172.21.15.18:0/4192093136 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17dc0ad170 con 0x7f17dc13f650 2024-03-29T11:16:06.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.278+0000 7f17d1ffb640 1 -- 172.21.15.18:0/4192093136 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f17d8001630 con 0x7f17dc13f650 2024-03-29T11:16:06.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.278+0000 7f17d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:06.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.278+0000 7f17d1ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:06.258 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.278+0000 7f17d1ffb640 10 monclient(hunting): dump: 2024-03-29T11:16:06.258 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:16:06.258 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:06.258 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:06.258 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:06.259 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:06.259 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:06.259 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:16:06.259 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:16:06.259 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:16:06.259 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:06.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.279+0000 7f17d1ffb640 1 -- 172.21.15.18:0/4192093136 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 767947812 0 0) 0x7f17dc0ad170 con 0x7f17dc13f650 2024-03-29T11:16:06.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.279+0000 7f17d1ffb640 1 -- 172.21.15.18:0/4192093136 >> v1:172.21.15.70:6789/0 conn(0x7f17dc0a9d70 legacy=0x7f17dc137dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:06.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.279+0000 7f17d1ffb640 1 -- 172.21.15.18:0/4192093136 >> v1:172.21.15.18:6789/0 conn(0x7f17dc1457f0 legacy=0x7f17dc1e2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:06.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.279+0000 7f17d1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:16:06.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.279+0000 7f17d1ffb640 1 monclient: found mon.c 2024-03-29T11:16:06.261 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.279+0000 7f17d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:06.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.279+0000 7f17d1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:16:06.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.279+0000 7f17d1ffb640 1 -- 172.21.15.18:0/4192093136 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17dc1384e0 con 0x7f17dc13f650 2024-03-29T11:16:06.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.279+0000 7f17d1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:16:06.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.279+0000 7f17d1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:16:06.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.279+0000 7f17d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:06.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.279+0000 7f17e3732640 5 monclient: authenticate success, global_id 4517 2024-03-29T11:16:06.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.279+0000 7f17e3732640 10 monclient: _renew_subs 2024-03-29T11:16:06.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.279+0000 7f17e3732640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:16:06.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.279+0000 7f17e3732640 1 -- 172.21.15.18:0/4192093136 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17dc138670 con 0x7f17dc13f650 2024-03-29T11:16:06.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.279+0000 7f17e3732640 10 client.4517.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:06.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.279+0000 7f17e3732640 10 monclient: _renew_subs 2024-03-29T11:16:06.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.279+0000 7f17e3732640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:16:06.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.279+0000 7f17e3732640 1 -- 172.21.15.18:0/4192093136 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f17dc13ba70 con 0x7f17dc13f650 2024-03-29T11:16:06.262 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.279+0000 7f17e3732640 1 librados: init done 2024-03-29T11:16:06.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.284+0000 7f17e3732640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:06.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.284+0000 7f17e3732640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:06.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.284+0000 7f17e3732640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:16:06.263 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.284+0000 7f17e3732640 1 -- 172.21.15.18:0/4192093136 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17dc075f10 con 0x7f17dc13f650 2024-03-29T11:16:06.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.284+0000 7f17d1ffb640 1 -- 172.21.15.18:0/4192093136 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17d8001250 con 0x7f17dc13f650 2024-03-29T11:16:06.264 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.284+0000 7f17d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:06.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.284+0000 7f17d1ffb640 1 -- 172.21.15.18:0/4192093136 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f17d8004ce0 con 0x7f17dc13f650 2024-03-29T11:16:06.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.284+0000 7f17d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:06.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.284+0000 7f17d1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:06.265 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.284+0000 7f17d1ffb640 10 monclient: dump: 2024-03-29T11:16:06.265 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:16:06.265 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:06.265 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:06.265 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:06.265 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:06.265 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:06.265 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:16:06.265 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:16:06.265 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:16:06.265 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:06.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.284+0000 7f17d1ffb640 1 -- 172.21.15.18:0/4192093136 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f17d801acb0 con 0x7f17dc13f650 2024-03-29T11:16:06.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.285+0000 7f17d1ffb640 1 -- 172.21.15.18:0/4192093136 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f17d8084bf0 con 0x7f17dc13f650 2024-03-29T11:16:06.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.285+0000 7f17d1ffb640 10 client.4517.objecter ms_dispatch 0x7f17dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:06.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.285+0000 7f17d1ffb640 3 client.4517.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:06.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.285+0000 7f17d1ffb640 3 client.4517.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:06.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.285+0000 7f17d1ffb640 20 client.4517.objecter dump_active .. 0 homeless 2024-03-29T11:16:06.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.284+0000 7f17d37fe640 4 set_mon_vals no callback set 2024-03-29T11:16:06.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.287+0000 7f17e14ce640 10 monclient: get_auth_request con 0x7f17c0076930 auth_method 0 2024-03-29T11:16:06.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.288+0000 7f17d1ffb640 10 client.4517.objecter ms_handle_connect 0x7f17c0076930 2024-03-29T11:16:06.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.292+0000 7f17d1ffb640 1 -- 172.21.15.18:0/4192093136 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f17d8050550 con 0x7f17dc13f650 2024-03-29T11:16:06.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.292+0000 7f17d1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:06.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.292+0000 7f17d1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:06.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.617+0000 7f17e3732640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:06.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.617+0000 7f17e3732640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:06.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.617+0000 7f17e3732640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:16:06.595 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.617+0000 7f17e3732640 1 -- 172.21.15.18:0/4192093136 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f17dc1d7190 con 0x7f17dc13f650 2024-03-29T11:16:06.596 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.619+0000 7f17d1ffb640 1 -- 172.21.15.18:0/4192093136 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f17d8052480 con 0x7f17dc13f650 2024-03-29T11:16:06.596 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.619+0000 7f17d1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:06.597 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.619+0000 7f17d1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:06.598 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:16:06.599 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:06.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.625+0000 7f17e3732640 10 librados: watch_flush enter 2024-03-29T11:16:06.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.625+0000 7f17e3732640 10 librados: watch_flush exit 2024-03-29T11:16:06.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.625+0000 7f17e3732640 20 client.4517.objecter shutdown clearing up homeless session... 2024-03-29T11:16:06.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.625+0000 7f17e3732640 10 client.4517.objecter successfully canceled tick 2024-03-29T11:16:06.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.625+0000 7f17e3732640 1 -- 172.21.15.18:0/4192093136 >> v1:172.21.15.70:6816/3101871562 conn(0x7f17c0076930 legacy=0x7f17c0078df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:06.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.625+0000 7f17e3732640 10 monclient: shutdown 2024-03-29T11:16:06.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.625+0000 7f17e3732640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:06.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.625+0000 7f17e3732640 1 -- 172.21.15.18:0/4192093136 >> v1:172.21.15.18:6790/0 conn(0x7f17dc13f650 legacy=0x7f17dc1d7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:06.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.625+0000 7f17e3732640 1 -- 172.21.15.18:0/4192093136 shutdown_connections 2024-03-29T11:16:06.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.625+0000 7f17e3732640 1 -- 172.21.15.18:0/4192093136 >> 172.21.15.18:0/4192093136 conn(0x7f17dc066a00 msgr2=0x7f17dc0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:06.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.625+0000 7f17e3732640 1 -- 172.21.15.18:0/4192093136 shutdown_connections 2024-03-29T11:16:06.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.626+0000 7f17e3732640 1 -- 172.21.15.18:0/4192093136 wait complete. 2024-03-29T11:16:06.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:06.626+0000 7f17e3732640 1 librados: shutdown 2024-03-29T11:16:06.623 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:13.430 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:13.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.599+0000 7fdfd2939640 10 monclient: get_monmap_and_config 2024-03-29T11:16:13.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.599+0000 7fdfd2939640 10 monclient: build_initial_monmap 2024-03-29T11:16:13.577 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.600+0000 7fdfd2939640 10 monclient: monmap: 2024-03-29T11:16:13.577 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:16:13.577 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:13.578 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:16:13.600884+0000 2024-03-29T11:16:13.578 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:16:13.600884+0000 2024-03-29T11:16:13.578 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:13.578 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:13.578 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:16:13.578 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:16:13.578 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:16:13.578 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:13.579 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.600+0000 7fdfd2939640 1 Processor -- start 2024-03-29T11:16:13.579 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.600+0000 7fdfd2939640 1 -- start start 2024-03-29T11:16:13.579 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.600+0000 7fdfd2939640 10 monclient: init 2024-03-29T11:16:13.579 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.602+0000 7fdfd2939640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:13.579 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.602+0000 7fdfd2939640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:16:13.579 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.602+0000 7fdfd2939640 10 monclient(hunting): picked mon.noname-c con 0x7fdfcc0b4790 addr v1:172.21.15.70:6789/0 2024-03-29T11:16:13.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.602+0000 7fdfd2939640 10 monclient(hunting): picked mon.noname-b con 0x7fdfcc0aea90 addr v1:172.21.15.18:6790/0 2024-03-29T11:16:13.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.602+0000 7fdfd2939640 10 monclient(hunting): picked mon.noname-a con 0x7fdfcc0b39f0 addr v1:172.21.15.18:6789/0 2024-03-29T11:16:13.580 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.603+0000 7fdfd2939640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfcc0b0c60 con 0x7fdfcc0b39f0 2024-03-29T11:16:13.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.603+0000 7fdfd2939640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfcc0af020 con 0x7fdfcc0b4790 2024-03-29T11:16:13.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.603+0000 7fdfd2939640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfcc13fbd0 con 0x7fdfcc0aea90 2024-03-29T11:16:13.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.603+0000 7fdfd2939640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:13.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.603+0000 7fdfd2939640 10 monclient(hunting): authenticate will time out at 1199.875119s 2024-03-29T11:16:13.581 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.604+0000 7fdfcb7fe640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fdfcc0aea90 0x7fdfcc0af540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:36060/0 (socket says 172.21.15.18:36060) 2024-03-29T11:16:13.582 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.604+0000 7fdfcb7fe640 1 -- 172.21.15.18:0/4094770410 learned_addr learned my addr 172.21.15.18:0/4094770410 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:16:13.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.605+0000 7fdfcaffd640 1 -- 172.21.15.18:0/4094770410 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 977596925 0 0) 0x7fdfcc13fbd0 con 0x7fdfcc0aea90 2024-03-29T11:16:13.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.605+0000 7fdfcaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:16:13.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.605+0000 7fdfcaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:13.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.605+0000 7fdfcaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:13.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.605+0000 7fdfcaffd640 10 monclient(hunting): my global_id is 4520 2024-03-29T11:16:13.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.605+0000 7fdfcaffd640 1 -- 172.21.15.18:0/4094770410 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfb4003520 con 0x7fdfcc0aea90 2024-03-29T11:16:13.583 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.606+0000 7fdfcaffd640 1 -- 172.21.15.18:0/4094770410 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2108660386 0 0) 0x7fdfb4003520 con 0x7fdfcc0aea90 2024-03-29T11:16:13.584 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.606+0000 7fdfcaffd640 1 -- 172.21.15.18:0/4094770410 >> v1:172.21.15.70:6789/0 conn(0x7fdfcc0b4790 legacy=0x7fdfcc0b1080 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:16:13.586 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.607+0000 7fdfcaffd640 1 -- 172.21.15.18:0/4094770410 >> v1:172.21.15.18:6789/0 conn(0x7fdfcc0b39f0 legacy=0x7fdfcc13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:16:13.586 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.607+0000 7fdfcaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:16:13.586 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.607+0000 7fdfcaffd640 1 monclient: found mon.noname-b 2024-03-29T11:16:13.586 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.607+0000 7fdfcaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:13.586 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.607+0000 7fdfcaffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:16:13.586 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.607+0000 7fdfcaffd640 1 -- 172.21.15.18:0/4094770410 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfcc13fd60 con 0x7fdfcc0aea90 2024-03-29T11:16:13.586 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.607+0000 7fdfcaffd640 10 monclient: _finish_auth 0 2024-03-29T11:16:13.587 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.607+0000 7fdfcaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:16:13.587 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.607+0000 7fdfcaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:13.587 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.608+0000 7fdfcaffd640 1 -- 172.21.15.18:0/4094770410 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fdfbc001690 con 0x7fdfcc0aea90 2024-03-29T11:16:13.587 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.608+0000 7fdfd2939640 5 monclient: authenticate success, global_id 4520 2024-03-29T11:16:13.587 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.608+0000 7fdfcaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:13.587 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.608+0000 7fdfcaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:16:13.587 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.608+0000 7fdfcaffd640 10 monclient: dump: 2024-03-29T11:16:13.587 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:16:13.587 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:13.587 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:13.587 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:13.587 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:13.587 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:13.587 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:16:13.588 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:16:13.588 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:16:13.588 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:13.588 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.608+0000 7fdfd2939640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:13.588 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.608+0000 7fdfcaffd640 1 -- 172.21.15.18:0/4094770410 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdfbc0018f0 con 0x7fdfcc0aea90 2024-03-29T11:16:13.588 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.608+0000 7fdfcaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:13.588 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.608+0000 7fdfcaffd640 1 -- 172.21.15.18:0/4094770410 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fdfbc001d40 con 0x7fdfcc0aea90 2024-03-29T11:16:13.588 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.608+0000 7fdfcaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:13.588 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.608+0000 7fdfcaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:13.588 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.608+0000 7fdfcaffd640 10 monclient: dump: 2024-03-29T11:16:13.588 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:16:13.588 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:13.588 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:13.588 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:13.588 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:13.588 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:13.589 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:16:13.589 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:16:13.589 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:16:13.589 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:13.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.608+0000 7fdfd2939640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:13.589 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.608+0000 7fdfd2939640 4 set_mon_vals no callback set 2024-03-29T11:16:13.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.609+0000 7fdfd2939640 10 monclient: shutdown 2024-03-29T11:16:13.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.609+0000 7fdfd2939640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:13.590 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.609+0000 7fdfd2939640 1 -- 172.21.15.18:0/4094770410 >> v1:172.21.15.18:6790/0 conn(0x7fdfcc0aea90 legacy=0x7fdfcc0af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:13.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.610+0000 7fdfd2939640 1 -- 172.21.15.18:0/4094770410 shutdown_connections 2024-03-29T11:16:13.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.610+0000 7fdfd2939640 1 -- 172.21.15.18:0/4094770410 >> 172.21.15.18:0/4094770410 conn(0x7fdfcc066a00 msgr2=0x7fdfcc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:13.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.610+0000 7fdfd2939640 1 -- 172.21.15.18:0/4094770410 shutdown_connections 2024-03-29T11:16:13.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.610+0000 7fdfd2939640 1 -- 172.21.15.18:0/4094770410 wait complete. 2024-03-29T11:16:13.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.611+0000 7fdfd2939640 10 monclient: build_initial_monmap 2024-03-29T11:16:13.591 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.611+0000 7fdfd2939640 10 monclient: monmap: 2024-03-29T11:16:13.591 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:16:13.591 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:13.591 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:16:13.591 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:16:13.591 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:13.591 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:13.591 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:16:13.591 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:16:13.592 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:16:13.592 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:13.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.612+0000 7fdfd2939640 1 librados: starting msgr at 2024-03-29T11:16:13.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.612+0000 7fdfd2939640 1 librados: starting objecter 2024-03-29T11:16:13.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.612+0000 7fdfd2939640 1 Processor -- start 2024-03-29T11:16:13.592 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.615+0000 7fdfd2939640 1 -- start start 2024-03-29T11:16:13.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.615+0000 7fdfd2939640 1 librados: setting wanted keys 2024-03-29T11:16:13.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.615+0000 7fdfd2939640 1 librados: calling monclient init 2024-03-29T11:16:13.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.615+0000 7fdfd2939640 10 monclient: init 2024-03-29T11:16:13.593 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.616+0000 7fdfd2939640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:13.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.616+0000 7fdfd2939640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:16:13.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.616+0000 7fdfd2939640 10 monclient(hunting): picked mon.noname-c con 0x7fdfcc0b39f0 addr v1:172.21.15.18:6790/0 2024-03-29T11:16:13.594 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.616+0000 7fdfd2939640 10 monclient(hunting): picked mon.noname-a con 0x7fdfcc0b4790 addr v1:172.21.15.18:6789/0 2024-03-29T11:16:13.597 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.619+0000 7fdfd2939640 10 monclient(hunting): picked mon.noname-b con 0x7fdfcc1354a0 addr v1:172.21.15.70:6789/0 2024-03-29T11:16:13.597 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.619+0000 7fdfd2939640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfcc1328d0 con 0x7fdfcc0b4790 2024-03-29T11:16:13.597 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.619+0000 7fdfd2939640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfcc133640 con 0x7fdfcc1354a0 2024-03-29T11:16:13.597 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.619+0000 7fdfd2939640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfcc1daf50 con 0x7fdfcc0b39f0 2024-03-29T11:16:13.597 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.619+0000 7fdfd2939640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:13.597 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.619+0000 7fdfd2939640 10 monclient(hunting): authenticate will time out at 1199.891718s 2024-03-29T11:16:13.597 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.619+0000 7fdfcbfff640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fdfcc0b39f0 0x7fdfcc1da840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:38892/0 (socket says 172.21.15.18:38892) 2024-03-29T11:16:13.597 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.619+0000 7fdfcbfff640 1 -- 172.21.15.18:0/205238651 learned_addr learned my addr 172.21.15.18:0/205238651 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:16:13.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.621+0000 7fdfc8ff9640 10 client.?.objecter ms_handle_connect 0x7fdfcc1354a0 2024-03-29T11:16:13.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.621+0000 7fdfc8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:13.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.621+0000 7fdfc8ff9640 1 -- 172.21.15.18:0/205238651 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2450046306 0 0) 0x7fdfcc133640 con 0x7fdfcc1354a0 2024-03-29T11:16:13.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.621+0000 7fdfc8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:16:13.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.621+0000 7fdfc8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:13.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.621+0000 7fdfc8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:13.603 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.621+0000 7fdfc8ff9640 10 monclient(hunting): my global_id is 4438 2024-03-29T11:16:13.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.621+0000 7fdfc8ff9640 1 -- 172.21.15.18:0/205238651 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfb8004eb0 con 0x7fdfcc1354a0 2024-03-29T11:16:13.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.621+0000 7fdfc8ff9640 10 client.?.objecter ms_handle_connect 0x7fdfcc0b39f0 2024-03-29T11:16:13.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.621+0000 7fdfc8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:13.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.622+0000 7fdfc8ff9640 10 client.?.objecter ms_handle_connect 0x7fdfcc0b4790 2024-03-29T11:16:13.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.622+0000 7fdfc8ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:13.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.622+0000 7fdfc8ff9640 1 -- 172.21.15.18:0/205238651 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 190721110 0 0) 0x7fdfcc1328d0 con 0x7fdfcc0b4790 2024-03-29T11:16:13.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.622+0000 7fdfc8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:16:13.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.622+0000 7fdfc8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:13.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.622+0000 7fdfc8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:13.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.622+0000 7fdfc8ff9640 10 monclient(hunting): my global_id is 4461 2024-03-29T11:16:13.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.622+0000 7fdfc8ff9640 1 -- 172.21.15.18:0/205238651 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfcc133640 con 0x7fdfcc0b4790 2024-03-29T11:16:13.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.622+0000 7fdfc8ff9640 1 -- 172.21.15.18:0/205238651 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 526771426 0 0) 0x7fdfcc133640 con 0x7fdfcc0b4790 2024-03-29T11:16:13.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.622+0000 7fdfc8ff9640 1 -- 172.21.15.18:0/205238651 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdfcc1328d0 con 0x7fdfcc0b4790 2024-03-29T11:16:13.604 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.622+0000 7fdfc8ff9640 1 -- 172.21.15.18:0/205238651 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fdfbc001650 con 0x7fdfcc0b4790 2024-03-29T11:16:13.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.622+0000 7fdfc8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:13.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.622+0000 7fdfc8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:16:13.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.622+0000 7fdfc8ff9640 10 monclient(hunting): dump: 2024-03-29T11:16:13.605 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:16:13.605 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:13.605 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:13.605 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:13.605 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:13.605 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:13.605 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:16:13.605 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:16:13.605 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:16:13.605 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:13.605 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.622+0000 7fdfc8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:13.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.622+0000 7fdfc8ff9640 1 -- 172.21.15.18:0/205238651 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1145970538 0 0) 0x7fdfcc1daf50 con 0x7fdfcc0b39f0 2024-03-29T11:16:13.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.622+0000 7fdfc8ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:16:13.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.622+0000 7fdfc8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:13.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.622+0000 7fdfc8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:13.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.622+0000 7fdfc8ff9640 10 monclient(hunting): my global_id is 4523 2024-03-29T11:16:13.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.622+0000 7fdfc8ff9640 1 -- 172.21.15.18:0/205238651 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfcc133640 con 0x7fdfcc0b39f0 2024-03-29T11:16:13.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.623+0000 7fdfc8ff9640 1 -- 172.21.15.18:0/205238651 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3099521130 0 0) 0x7fdfb8004eb0 con 0x7fdfcc1354a0 2024-03-29T11:16:13.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.623+0000 7fdfc8ff9640 1 -- 172.21.15.18:0/205238651 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdfcc1daf50 con 0x7fdfcc1354a0 2024-03-29T11:16:13.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.623+0000 7fdfc8ff9640 1 -- 172.21.15.18:0/205238651 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fdfc4001890 con 0x7fdfcc1354a0 2024-03-29T11:16:13.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.623+0000 7fdfc8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:13.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.623+0000 7fdfc8ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:16:13.606 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.623+0000 7fdfc8ff9640 10 monclient(hunting): dump: 2024-03-29T11:16:13.606 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:16:13.607 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:13.607 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:13.607 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:13.607 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:13.607 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:13.607 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:16:13.607 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:16:13.607 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:16:13.607 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:13.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.623+0000 7fdfc8ff9640 1 -- 172.21.15.18:0/205238651 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 159960528 0 0) 0x7fdfcc133640 con 0x7fdfcc0b39f0 2024-03-29T11:16:13.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.623+0000 7fdfc8ff9640 1 -- 172.21.15.18:0/205238651 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdfb8004eb0 con 0x7fdfcc0b39f0 2024-03-29T11:16:13.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.623+0000 7fdfc8ff9640 1 -- 172.21.15.18:0/205238651 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fdfc0003f20 con 0x7fdfcc0b39f0 2024-03-29T11:16:13.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.623+0000 7fdfc8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:13.607 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.623+0000 7fdfc8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:13.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.623+0000 7fdfc8ff9640 10 monclient(hunting): dump: 2024-03-29T11:16:13.608 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:16:13.608 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:13.608 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:13.608 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:13.608 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:13.608 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:13.608 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:16:13.608 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:16:13.608 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:16:13.608 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:13.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.623+0000 7fdfc8ff9640 1 -- 172.21.15.18:0/205238651 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1632370953 0 0) 0x7fdfcc1328d0 con 0x7fdfcc0b4790 2024-03-29T11:16:13.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.623+0000 7fdfc8ff9640 1 -- 172.21.15.18:0/205238651 >> v1:172.21.15.18:6790/0 conn(0x7fdfcc0b39f0 legacy=0x7fdfcc1da840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:13.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.623+0000 7fdfc8ff9640 1 -- 172.21.15.18:0/205238651 >> v1:172.21.15.70:6789/0 conn(0x7fdfcc1354a0 legacy=0x7fdfcc1e2b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:13.608 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.623+0000 7fdfc8ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:16:13.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.623+0000 7fdfc8ff9640 1 monclient: found mon.a 2024-03-29T11:16:13.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.623+0000 7fdfc8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:13.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.623+0000 7fdfc8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:16:13.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.623+0000 7fdfc8ff9640 1 -- 172.21.15.18:0/205238651 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfcc133060 con 0x7fdfcc0b4790 2024-03-29T11:16:13.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.623+0000 7fdfc8ff9640 10 monclient: _finish_auth 0 2024-03-29T11:16:13.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.623+0000 7fdfc8ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:16:13.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.623+0000 7fdfc8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:13.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.623+0000 7fdfd2939640 5 monclient: authenticate success, global_id 4461 2024-03-29T11:16:13.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.623+0000 7fdfd2939640 10 monclient: _renew_subs 2024-03-29T11:16:13.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.623+0000 7fdfd2939640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:16:13.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.623+0000 7fdfd2939640 1 -- 172.21.15.18:0/205238651 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfcc133310 con 0x7fdfcc0b4790 2024-03-29T11:16:13.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.623+0000 7fdfd2939640 10 client.4461.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:13.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.623+0000 7fdfd2939640 10 monclient: _renew_subs 2024-03-29T11:16:13.609 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.623+0000 7fdfd2939640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:16:13.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.623+0000 7fdfd2939640 1 -- 172.21.15.18:0/205238651 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfcc12d410 con 0x7fdfcc0b4790 2024-03-29T11:16:13.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.623+0000 7fdfd2939640 1 librados: init done 2024-03-29T11:16:13.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.625+0000 7fdfd2939640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:13.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.625+0000 7fdfd2939640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:13.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.625+0000 7fdfd2939640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:16:13.610 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.625+0000 7fdfd2939640 1 -- 172.21.15.18:0/205238651 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfcc134dc0 con 0x7fdfcc0b4790 2024-03-29T11:16:13.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.627+0000 7fdfc8ff9640 1 -- 172.21.15.18:0/205238651 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdfbc001020 con 0x7fdfcc0b4790 2024-03-29T11:16:13.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.627+0000 7fdfc8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:13.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.627+0000 7fdfc8ff9640 1 -- 172.21.15.18:0/205238651 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fdfbc004c70 con 0x7fdfcc0b4790 2024-03-29T11:16:13.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.627+0000 7fdfc8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:13.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.627+0000 7fdfc8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:16:13.611 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.627+0000 7fdfc8ff9640 10 monclient: dump: 2024-03-29T11:16:13.611 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:16:13.611 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:13.611 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:13.612 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:13.612 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:13.612 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:13.612 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:16:13.612 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:16:13.612 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:16:13.612 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:13.612 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.627+0000 7fdfc8ff9640 1 -- 172.21.15.18:0/205238651 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7fdfbc01ac40 con 0x7fdfcc0b4790 2024-03-29T11:16:13.612 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.628+0000 7fdfc8ff9640 1 -- 172.21.15.18:0/205238651 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7fdfbc084b80 con 0x7fdfcc0b4790 2024-03-29T11:16:13.612 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.628+0000 7fdfc8ff9640 10 client.4461.objecter ms_dispatch 0x7fdfcc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:13.612 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.628+0000 7fdfc8ff9640 3 client.4461.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:13.613 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.628+0000 7fdfc8ff9640 3 client.4461.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:13.613 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.628+0000 7fdfc8ff9640 20 client.4461.objecter dump_active .. 0 homeless 2024-03-29T11:16:13.613 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.628+0000 7fdfca7fc640 4 set_mon_vals no callback set 2024-03-29T11:16:13.613 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.628+0000 7fdfcbfff640 10 monclient: get_auth_request con 0x7fdfb806e200 auth_method 0 2024-03-29T11:16:13.613 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.628+0000 7fdfc8ff9640 10 client.4461.objecter ms_handle_connect 0x7fdfb806e200 2024-03-29T11:16:13.613 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.634+0000 7fdfc8ff9640 1 -- 172.21.15.18:0/205238651 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdfbc0504e0 con 0x7fdfcc0b4790 2024-03-29T11:16:13.613 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.634+0000 7fdfc8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:13.613 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.634+0000 7fdfc8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:13.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.968+0000 7fdfd2939640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:13.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.968+0000 7fdfd2939640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:13.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.968+0000 7fdfd2939640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:16:13.946 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.968+0000 7fdfd2939640 1 -- 172.21.15.18:0/205238651 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdfcc075f10 con 0x7fdfcc0b4790 2024-03-29T11:16:13.946 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.969+0000 7fdfc8ff9640 1 -- 172.21.15.18:0/205238651 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7fdfbc0504e0 con 0x7fdfcc0b4790 2024-03-29T11:16:13.946 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.969+0000 7fdfc8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:13.946 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.969+0000 7fdfc8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:13.947 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:16:13.947 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:13.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.972+0000 7fdfd2939640 10 librados: watch_flush enter 2024-03-29T11:16:13.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.972+0000 7fdfd2939640 10 librados: watch_flush exit 2024-03-29T11:16:13.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.972+0000 7fdfd2939640 20 client.4461.objecter shutdown clearing up homeless session... 2024-03-29T11:16:13.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.972+0000 7fdfd2939640 10 client.4461.objecter successfully canceled tick 2024-03-29T11:16:13.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.973+0000 7fdfd2939640 1 -- 172.21.15.18:0/205238651 >> v1:172.21.15.70:6816/3101871562 conn(0x7fdfb806e200 legacy=0x7fdfb80706c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:13.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.973+0000 7fdfd2939640 10 monclient: shutdown 2024-03-29T11:16:13.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.973+0000 7fdfd2939640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:13.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.973+0000 7fdfd2939640 1 -- 172.21.15.18:0/205238651 >> v1:172.21.15.18:6789/0 conn(0x7fdfcc0b4790 legacy=0x7fdfcc12cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:13.952 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.974+0000 7fdfd2939640 1 -- 172.21.15.18:0/205238651 shutdown_connections 2024-03-29T11:16:13.952 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.974+0000 7fdfd2939640 1 -- 172.21.15.18:0/205238651 >> 172.21.15.18:0/205238651 conn(0x7fdfcc066a00 msgr2=0x7fdfcc0b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:13.952 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.975+0000 7fdfd2939640 1 -- 172.21.15.18:0/205238651 shutdown_connections 2024-03-29T11:16:13.952 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.975+0000 7fdfd2939640 1 -- 172.21.15.18:0/205238651 wait complete. 2024-03-29T11:16:13.952 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:13.975+0000 7fdfd2939640 1 librados: shutdown 2024-03-29T11:16:13.970 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:20.776 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:20.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.938+0000 7f53887db640 10 monclient: get_monmap_and_config 2024-03-29T11:16:20.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.938+0000 7f53887db640 10 monclient: build_initial_monmap 2024-03-29T11:16:20.915 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.938+0000 7f53887db640 10 monclient: monmap: 2024-03-29T11:16:20.915 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:16:20.915 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:20.916 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:16:20.938969+0000 2024-03-29T11:16:20.916 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:16:20.938969+0000 2024-03-29T11:16:20.916 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:20.916 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:20.916 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:16:20.916 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:16:20.916 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:16:20.916 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:20.918 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.938+0000 7f53887db640 1 Processor -- start 2024-03-29T11:16:20.918 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.939+0000 7f53887db640 1 -- start start 2024-03-29T11:16:20.918 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.939+0000 7f53887db640 10 monclient: init 2024-03-29T11:16:20.918 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.940+0000 7f53887db640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:20.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.940+0000 7f53887db640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:16:20.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.940+0000 7f53887db640 10 monclient(hunting): picked mon.noname-c con 0x7f53800b0860 addr v1:172.21.15.70:6789/0 2024-03-29T11:16:20.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.940+0000 7f53887db640 10 monclient(hunting): picked mon.noname-b con 0x7f538013ef30 addr v1:172.21.15.18:6790/0 2024-03-29T11:16:20.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.940+0000 7f53887db640 10 monclient(hunting): picked mon.noname-a con 0x7f53800af8e0 addr v1:172.21.15.18:6789/0 2024-03-29T11:16:20.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.940+0000 7f53887db640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53800b0c40 con 0x7f53800af8e0 2024-03-29T11:16:20.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.940+0000 7f53887db640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53800aed10 con 0x7f53800b0860 2024-03-29T11:16:20.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.940+0000 7f53887db640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53800afd50 con 0x7f538013ef30 2024-03-29T11:16:20.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.940+0000 7f53887db640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:20.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.940+0000 7f53887db640 10 monclient(hunting): authenticate will time out at 1207.212265s 2024-03-29T11:16:20.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.940+0000 7f5386d78640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f53800af8e0 0x7f5380149a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:41612/0 (socket says 172.21.15.18:41612) 2024-03-29T11:16:20.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.940+0000 7f5386d78640 1 -- 172.21.15.18:0/3375359359 learned_addr learned my addr 172.21.15.18:0/3375359359 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:16:20.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.941+0000 7f5385575640 1 -- 172.21.15.18:0/3375359359 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1394388892 0 0) 0x7f53800b0c40 con 0x7f53800af8e0 2024-03-29T11:16:20.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.941+0000 7f5385575640 10 monclient(hunting): _negotiate 2024-03-29T11:16:20.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.941+0000 7f5385575640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:20.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.941+0000 7f5385575640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:20.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.941+0000 7f5385575640 10 monclient(hunting): my global_id is 4464 2024-03-29T11:16:20.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.941+0000 7f5385575640 1 -- 172.21.15.18:0/3375359359 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5368003520 con 0x7f53800af8e0 2024-03-29T11:16:20.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.941+0000 7f5385575640 1 -- 172.21.15.18:0/3375359359 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1784584321 0 0) 0x7f53800afd50 con 0x7f538013ef30 2024-03-29T11:16:20.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.941+0000 7f5385575640 10 monclient(hunting): _negotiate 2024-03-29T11:16:20.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.941+0000 7f5385575640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:20.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.941+0000 7f5385575640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:20.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.941+0000 7f5385575640 10 monclient(hunting): my global_id is 4526 2024-03-29T11:16:20.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.941+0000 7f5385575640 1 -- 172.21.15.18:0/3375359359 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53800b0c40 con 0x7f538013ef30 2024-03-29T11:16:20.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.941+0000 7f5385575640 1 -- 172.21.15.18:0/3375359359 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3953974165 0 0) 0x7f5368003520 con 0x7f53800af8e0 2024-03-29T11:16:20.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.941+0000 7f5385575640 1 -- 172.21.15.18:0/3375359359 >> v1:172.21.15.18:6790/0 conn(0x7f538013ef30 legacy=0x7f53800af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:20.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.941+0000 7f5385575640 1 -- 172.21.15.18:0/3375359359 >> v1:172.21.15.70:6789/0 conn(0x7f53800b0860 legacy=0x7f538013e820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:16:20.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.942+0000 7f5385575640 10 monclient: _finish_hunting 0 2024-03-29T11:16:20.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.942+0000 7f5385575640 1 monclient: found mon.noname-a 2024-03-29T11:16:20.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.942+0000 7f5385575640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:20.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.942+0000 7f5385575640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-29T11:16:20.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.942+0000 7f5385575640 1 -- 172.21.15.18:0/3375359359 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53800ab1b0 con 0x7f53800af8e0 2024-03-29T11:16:20.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.942+0000 7f5385575640 10 monclient: _finish_auth 0 2024-03-29T11:16:20.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.942+0000 7f5385575640 10 monclient: _check_auth_tickets 2024-03-29T11:16:20.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.942+0000 7f5385575640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:20.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.942+0000 7f53887db640 5 monclient: authenticate success, global_id 4464 2024-03-29T11:16:20.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.942+0000 7f53887db640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:20.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.942+0000 7f5385575640 1 -- 172.21.15.18:0/3375359359 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f5374001690 con 0x7f53800af8e0 2024-03-29T11:16:20.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.942+0000 7f5385575640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:20.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.942+0000 7f5385575640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:16:20.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.942+0000 7f5385575640 10 monclient: dump: 2024-03-29T11:16:20.923 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:16:20.923 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:20.923 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:20.924 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:20.924 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:20.924 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:20.924 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:16:20.924 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:16:20.924 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:16:20.924 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:20.924 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.942+0000 7f53887db640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:20.924 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.943+0000 7f5385575640 1 -- 172.21.15.18:0/3375359359 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53740018b0 con 0x7f53800af8e0 2024-03-29T11:16:20.924 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.943+0000 7f5385575640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:20.924 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.943+0000 7f53887db640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:20.924 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.943+0000 7f53887db640 4 set_mon_vals no callback set 2024-03-29T11:16:20.924 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.943+0000 7f5385575640 1 -- 172.21.15.18:0/3375359359 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f5374001c40 con 0x7f53800af8e0 2024-03-29T11:16:20.924 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.943+0000 7f5385575640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:20.924 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.943+0000 7f5385575640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:16:20.924 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.943+0000 7f5385575640 10 monclient: dump: 2024-03-29T11:16:20.925 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:16:20.925 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:20.925 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:20.925 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:20.925 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:20.925 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:20.925 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:16:20.925 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:16:20.925 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:16:20.925 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:20.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.943+0000 7f53887db640 10 monclient: shutdown 2024-03-29T11:16:20.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.944+0000 7f53887db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:20.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.944+0000 7f53887db640 1 -- 172.21.15.18:0/3375359359 >> v1:172.21.15.18:6789/0 conn(0x7f53800af8e0 legacy=0x7f5380149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:20.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.944+0000 7f53887db640 1 -- 172.21.15.18:0/3375359359 shutdown_connections 2024-03-29T11:16:20.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.944+0000 7f53887db640 1 -- 172.21.15.18:0/3375359359 >> 172.21.15.18:0/3375359359 conn(0x7f538006b660 msgr2=0x7f538006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:20.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.944+0000 7f53887db640 1 -- 172.21.15.18:0/3375359359 shutdown_connections 2024-03-29T11:16:20.926 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.944+0000 7f53887db640 1 -- 172.21.15.18:0/3375359359 wait complete. 2024-03-29T11:16:20.926 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.945+0000 7f53887db640 10 monclient: build_initial_monmap 2024-03-29T11:16:20.926 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.945+0000 7f53887db640 10 monclient: monmap: 2024-03-29T11:16:20.926 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:16:20.926 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:20.926 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:16:20.926 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:16:20.926 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:20.926 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:20.926 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:16:20.926 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:16:20.926 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:16:20.926 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:20.926 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.945+0000 7f53887db640 1 librados: starting msgr at 2024-03-29T11:16:20.926 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.945+0000 7f53887db640 1 librados: starting objecter 2024-03-29T11:16:20.927 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.945+0000 7f53887db640 1 Processor -- start 2024-03-29T11:16:20.927 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.945+0000 7f53887db640 1 -- start start 2024-03-29T11:16:20.927 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.945+0000 7f53887db640 1 librados: setting wanted keys 2024-03-29T11:16:20.927 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.945+0000 7f53887db640 1 librados: calling monclient init 2024-03-29T11:16:20.927 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.945+0000 7f53887db640 10 monclient: init 2024-03-29T11:16:20.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.946+0000 7f53887db640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:20.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.946+0000 7f53887db640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:16:20.934 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.946+0000 7f53887db640 10 monclient(hunting): picked mon.noname-c con 0x7f53800af8e0 addr v1:172.21.15.18:6790/0 2024-03-29T11:16:20.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.946+0000 7f53887db640 10 monclient(hunting): picked mon.noname-a con 0x7f53800b0860 addr v1:172.21.15.18:6789/0 2024-03-29T11:16:20.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.946+0000 7f5386577640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f53800af8e0 0x7f53801dabd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:57844/0 (socket says 172.21.15.18:57844) 2024-03-29T11:16:20.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.946+0000 7f5386577640 1 -- 172.21.15.18:0/961179097 learned_addr learned my addr 172.21.15.18:0/961179097 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:16:20.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.946+0000 7f537effd640 10 client.?.objecter ms_handle_connect 0x7f53800b0860 2024-03-29T11:16:20.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.946+0000 7f537effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:20.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.947+0000 7f537effd640 10 client.?.objecter ms_handle_connect 0x7f53800af8e0 2024-03-29T11:16:20.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.947+0000 7f537effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:20.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.947+0000 7f53887db640 10 monclient(hunting): picked mon.noname-b con 0x7f538013ef30 addr v1:172.21.15.70:6789/0 2024-03-29T11:16:20.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.947+0000 7f53887db640 1 -- 172.21.15.18:0/961179097 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53801da740 con 0x7f53800b0860 2024-03-29T11:16:20.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.947+0000 7f53887db640 1 -- 172.21.15.18:0/961179097 --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53801c5250 con 0x7f538013ef30 2024-03-29T11:16:20.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.947+0000 7f53887db640 1 -- 172.21.15.18:0/961179097 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5380133150 con 0x7f53800af8e0 2024-03-29T11:16:20.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.947+0000 7f537effd640 1 -- 172.21.15.18:0/961179097 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4175466873 0 0) 0x7f53801da740 con 0x7f53800b0860 2024-03-29T11:16:20.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.947+0000 7f53887db640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:20.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.947+0000 7f53887db640 10 monclient(hunting): authenticate will time out at 1207.219480s 2024-03-29T11:16:20.935 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.947+0000 7f537effd640 10 monclient(hunting): _negotiate 2024-03-29T11:16:20.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.947+0000 7f537effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:20.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.947+0000 7f537effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:20.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.947+0000 7f537effd640 10 monclient(hunting): my global_id is 4467 2024-03-29T11:16:20.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.947+0000 7f537effd640 1 -- 172.21.15.18:0/961179097 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f536c004eb0 con 0x7f53800b0860 2024-03-29T11:16:20.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.950+0000 7f537effd640 1 -- 172.21.15.18:0/961179097 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1154826830 0 0) 0x7f5380133150 con 0x7f53800af8e0 2024-03-29T11:16:20.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.950+0000 7f537effd640 10 monclient(hunting): _negotiate 2024-03-29T11:16:20.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.950+0000 7f537effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:20.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.950+0000 7f537effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:20.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.950+0000 7f537effd640 10 monclient(hunting): my global_id is 4529 2024-03-29T11:16:20.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.950+0000 7f537effd640 1 -- 172.21.15.18:0/961179097 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53801da740 con 0x7f53800af8e0 2024-03-29T11:16:20.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.950+0000 7f537effd640 1 -- 172.21.15.18:0/961179097 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2618639313 0 0) 0x7f536c004eb0 con 0x7f53800b0860 2024-03-29T11:16:20.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.950+0000 7f537effd640 1 -- 172.21.15.18:0/961179097 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5380133150 con 0x7f53800b0860 2024-03-29T11:16:20.936 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.950+0000 7f537effd640 1 -- 172.21.15.18:0/961179097 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f5378001b40 con 0x7f53800b0860 2024-03-29T11:16:20.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.950+0000 7f537effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:20.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.950+0000 7f537effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:16:20.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.950+0000 7f537effd640 10 monclient(hunting): dump: 2024-03-29T11:16:20.937 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:16:20.937 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:20.937 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:20.937 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:20.937 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:20.937 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:20.937 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:16:20.937 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:16:20.937 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:16:20.937 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:20.937 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.950+0000 7f537effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:20.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.951+0000 7f537effd640 1 -- 172.21.15.18:0/961179097 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4069845980 0 0) 0x7f5380133150 con 0x7f53800b0860 2024-03-29T11:16:20.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.951+0000 7f537effd640 1 -- 172.21.15.18:0/961179097 >> v1:172.21.15.18:6790/0 conn(0x7f53800af8e0 legacy=0x7f53801dabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:20.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.951+0000 7f537effd640 1 -- 172.21.15.18:0/961179097 >> v1:172.21.15.70:6789/0 conn(0x7f538013ef30 legacy=0x7f5380132a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:20.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.951+0000 7f537effd640 10 monclient: _finish_hunting 0 2024-03-29T11:16:20.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.951+0000 7f537effd640 1 monclient: found mon.a 2024-03-29T11:16:20.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.951+0000 7f537effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:20.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.951+0000 7f537effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:16:20.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.951+0000 7f537effd640 1 -- 172.21.15.18:0/961179097 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53800b11d0 con 0x7f53800b0860 2024-03-29T11:16:20.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.951+0000 7f537effd640 10 monclient: _finish_auth 0 2024-03-29T11:16:20.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.951+0000 7f537effd640 10 monclient: _check_auth_tickets 2024-03-29T11:16:20.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.951+0000 7f537effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:20.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.951+0000 7f53887db640 5 monclient: authenticate success, global_id 4467 2024-03-29T11:16:20.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.951+0000 7f53887db640 10 monclient: _renew_subs 2024-03-29T11:16:20.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.951+0000 7f53887db640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:16:20.939 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.951+0000 7f53887db640 1 -- 172.21.15.18:0/961179097 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53801c54c0 con 0x7f53800b0860 2024-03-29T11:16:20.939 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.951+0000 7f537effd640 1 -- 172.21.15.18:0/961179097 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5378002030 con 0x7f53800b0860 2024-03-29T11:16:20.939 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.951+0000 7f53887db640 10 client.4467.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:20.939 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.951+0000 7f53887db640 10 monclient: _renew_subs 2024-03-29T11:16:20.939 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.951+0000 7f53887db640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:16:20.939 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.951+0000 7f53887db640 1 -- 172.21.15.18:0/961179097 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f53801db4c0 con 0x7f53800b0860 2024-03-29T11:16:20.939 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.952+0000 7f53887db640 1 librados: init done 2024-03-29T11:16:20.939 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.952+0000 7f537effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:20.939 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.952+0000 7f5384d74640 4 set_mon_vals no callback set 2024-03-29T11:16:20.939 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.953+0000 7f537effd640 1 -- 172.21.15.18:0/961179097 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f5378003b10 con 0x7f53800b0860 2024-03-29T11:16:20.939 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.953+0000 7f537effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:20.939 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.953+0000 7f537effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:16:20.939 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.953+0000 7f537effd640 10 monclient: dump: 2024-03-29T11:16:20.939 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:16:20.939 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:20.940 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:20.940 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:20.940 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:20.940 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:20.940 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:16:20.940 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:16:20.940 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:16:20.940 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:20.940 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.953+0000 7f53887db640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:20.940 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.953+0000 7f53887db640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:20.940 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.953+0000 7f53887db640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:16:20.940 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.953+0000 7f53887db640 1 -- 172.21.15.18:0/961179097 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5360005080 con 0x7f53800b0860 2024-03-29T11:16:20.940 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.954+0000 7f537effd640 1 -- 172.21.15.18:0/961179097 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f5378019ae0 con 0x7f53800b0860 2024-03-29T11:16:20.940 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.954+0000 7f537effd640 1 -- 172.21.15.18:0/961179097 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f5378083af0 con 0x7f53800b0860 2024-03-29T11:16:20.941 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.954+0000 7f537effd640 10 client.4467.objecter ms_dispatch 0x7f5380000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:20.941 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.954+0000 7f537effd640 3 client.4467.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:20.941 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.954+0000 7f537effd640 3 client.4467.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:20.941 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.954+0000 7f537effd640 20 client.4467.objecter dump_active .. 0 homeless 2024-03-29T11:16:20.941 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.955+0000 7f5386577640 10 monclient: get_auth_request con 0x7f536c06dce0 auth_method 0 2024-03-29T11:16:20.941 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.955+0000 7f537effd640 10 client.4467.objecter ms_handle_connect 0x7f536c06dce0 2024-03-29T11:16:20.942 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.962+0000 7f537effd640 1 -- 172.21.15.18:0/961179097 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f537804f480 con 0x7f53800b0860 2024-03-29T11:16:20.942 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.962+0000 7f537effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:20.942 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:20.962+0000 7f537effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:21.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:21.296+0000 7f53887db640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:21.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:21.296+0000 7f53887db640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:21.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:21.297+0000 7f53887db640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:16:21.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:21.297+0000 7f53887db640 1 -- 172.21.15.18:0/961179097 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5360005330 con 0x7f53800b0860 2024-03-29T11:16:21.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:21.298+0000 7f537effd640 1 -- 172.21.15.18:0/961179097 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f53780513b0 con 0x7f53800b0860 2024-03-29T11:16:21.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:21.298+0000 7f537effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:21.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:21.298+0000 7f537effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:21.276 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:16:21.276 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:21.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:21.302+0000 7f53887db640 10 librados: watch_flush enter 2024-03-29T11:16:21.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:21.303+0000 7f53887db640 10 librados: watch_flush exit 2024-03-29T11:16:21.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:21.303+0000 7f53887db640 20 client.4467.objecter shutdown clearing up homeless session... 2024-03-29T11:16:21.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:21.303+0000 7f53887db640 10 client.4467.objecter successfully canceled tick 2024-03-29T11:16:21.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:21.303+0000 7f53887db640 1 -- 172.21.15.18:0/961179097 >> v1:172.21.15.70:6816/3101871562 conn(0x7f536c06dce0 legacy=0x7f536c0701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:21.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:21.303+0000 7f53887db640 10 monclient: shutdown 2024-03-29T11:16:21.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:21.303+0000 7f53887db640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:21.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:21.303+0000 7f53887db640 1 -- 172.21.15.18:0/961179097 >> v1:172.21.15.18:6789/0 conn(0x7f53800b0860 legacy=0x7f5380132330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:21.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:21.303+0000 7f53887db640 1 -- 172.21.15.18:0/961179097 shutdown_connections 2024-03-29T11:16:21.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:21.303+0000 7f53887db640 1 -- 172.21.15.18:0/961179097 >> 172.21.15.18:0/961179097 conn(0x7f538006b660 msgr2=0x7f538013e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:21.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:21.304+0000 7f53887db640 1 -- 172.21.15.18:0/961179097 shutdown_connections 2024-03-29T11:16:21.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:21.304+0000 7f53887db640 1 -- 172.21.15.18:0/961179097 wait complete. 2024-03-29T11:16:21.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:21.304+0000 7f53887db640 1 librados: shutdown 2024-03-29T11:16:21.300 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:28.106 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:28.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.274+0000 7fdabc235640 10 monclient: get_monmap_and_config 2024-03-29T11:16:28.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.274+0000 7fdabc235640 10 monclient: build_initial_monmap 2024-03-29T11:16:28.252 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.274+0000 7fdabc235640 10 monclient: monmap: 2024-03-29T11:16:28.252 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:16:28.252 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:28.252 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:16:28.275602+0000 2024-03-29T11:16:28.252 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:16:28.275602+0000 2024-03-29T11:16:28.252 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:28.252 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:28.253 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:16:28.253 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:16:28.253 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:16:28.253 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:28.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.275+0000 7fdabc235640 1 Processor -- start 2024-03-29T11:16:28.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.276+0000 7fdabc235640 1 -- start start 2024-03-29T11:16:28.253 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.276+0000 7fdabc235640 10 monclient: init 2024-03-29T11:16:28.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.276+0000 7fdabc235640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:28.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.276+0000 7fdabc235640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:16:28.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.276+0000 7fdabc235640 10 monclient(hunting): picked mon.noname-a con 0x7fdab41457f0 addr v1:172.21.15.18:6789/0 2024-03-29T11:16:28.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.276+0000 7fdabc235640 10 monclient(hunting): picked mon.noname-c con 0x7fdab413f650 addr v1:172.21.15.70:6789/0 2024-03-29T11:16:28.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.276+0000 7fdabc235640 10 monclient(hunting): picked mon.noname-b con 0x7fdab40a9d70 addr v1:172.21.15.18:6790/0 2024-03-29T11:16:28.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.276+0000 7fdabc235640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab40aaef0 con 0x7fdab41457f0 2024-03-29T11:16:28.254 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.276+0000 7fdabc235640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab4145bd0 con 0x7fdab413f650 2024-03-29T11:16:28.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.277+0000 7fdabc235640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab4145eb0 con 0x7fdab40a9d70 2024-03-29T11:16:28.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.277+0000 7fdabc235640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:28.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.277+0000 7fdabc235640 10 monclient(hunting): authenticate will time out at 1214.548932s 2024-03-29T11:16:28.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.277+0000 7fdab9fd1640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fdab41457f0 0x7fdab413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:47442/0 (socket says 172.21.15.18:47442) 2024-03-29T11:16:28.255 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.277+0000 7fdab9fd1640 1 -- 172.21.15.18:0/1042874451 learned_addr learned my addr 172.21.15.18:0/1042874451 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:16:28.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.277+0000 7fdab8fcf640 1 -- 172.21.15.18:0/1042874451 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 407121331 0 0) 0x7fdab40aaef0 con 0x7fdab41457f0 2024-03-29T11:16:28.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.277+0000 7fdab8fcf640 10 monclient(hunting): _negotiate 2024-03-29T11:16:28.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.277+0000 7fdab8fcf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:28.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.277+0000 7fdab8fcf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:28.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.277+0000 7fdab8fcf640 10 monclient(hunting): my global_id is 4470 2024-03-29T11:16:28.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.277+0000 7fdab8fcf640 1 -- 172.21.15.18:0/1042874451 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaa0003520 con 0x7fdab41457f0 2024-03-29T11:16:28.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.278+0000 7fdab8fcf640 1 -- 172.21.15.18:0/1042874451 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3625203733 0 0) 0x7fdab4145eb0 con 0x7fdab40a9d70 2024-03-29T11:16:28.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.278+0000 7fdab8fcf640 10 monclient(hunting): _negotiate 2024-03-29T11:16:28.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.278+0000 7fdab8fcf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:28.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.278+0000 7fdab8fcf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:28.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.278+0000 7fdab8fcf640 10 monclient(hunting): my global_id is 4532 2024-03-29T11:16:28.266 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.278+0000 7fdab8fcf640 1 -- 172.21.15.18:0/1042874451 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdab40aaef0 con 0x7fdab40a9d70 2024-03-29T11:16:28.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.278+0000 7fdab8fcf640 1 -- 172.21.15.18:0/1042874451 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1914397076 0 0) 0x7fdab4145bd0 con 0x7fdab413f650 2024-03-29T11:16:28.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.278+0000 7fdab8fcf640 10 monclient(hunting): _negotiate 2024-03-29T11:16:28.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.278+0000 7fdab8fcf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:28.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.278+0000 7fdab8fcf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:28.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.278+0000 7fdab8fcf640 10 monclient(hunting): my global_id is 4441 2024-03-29T11:16:28.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.278+0000 7fdab8fcf640 1 -- 172.21.15.18:0/1042874451 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdab4145eb0 con 0x7fdab413f650 2024-03-29T11:16:28.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.278+0000 7fdab8fcf640 1 -- 172.21.15.18:0/1042874451 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1242204868 0 0) 0x7fdab4145eb0 con 0x7fdab413f650 2024-03-29T11:16:28.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.278+0000 7fdab8fcf640 1 -- 172.21.15.18:0/1042874451 >> v1:172.21.15.18:6790/0 conn(0x7fdab40a9d70 legacy=0x7fdab414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:28.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.278+0000 7fdab8fcf640 1 -- 172.21.15.18:0/1042874451 >> v1:172.21.15.18:6789/0 conn(0x7fdab41457f0 legacy=0x7fdab413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:28.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.278+0000 7fdab8fcf640 10 monclient: _finish_hunting 0 2024-03-29T11:16:28.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.278+0000 7fdab8fcf640 1 monclient: found mon.noname-c 2024-03-29T11:16:28.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.278+0000 7fdab8fcf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:28.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.278+0000 7fdab8fcf640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.70:6789/0 2024-03-29T11:16:28.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.278+0000 7fdab8fcf640 1 -- 172.21.15.18:0/1042874451 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab4146340 con 0x7fdab413f650 2024-03-29T11:16:28.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.278+0000 7fdab8fcf640 10 monclient: _finish_auth 0 2024-03-29T11:16:28.267 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.278+0000 7fdab8fcf640 10 monclient: _check_auth_tickets 2024-03-29T11:16:28.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.278+0000 7fdab8fcf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:28.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.278+0000 7fdab8fcf640 1 -- 172.21.15.18:0/1042874451 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fdab00015b0 con 0x7fdab413f650 2024-03-29T11:16:28.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.278+0000 7fdab8fcf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:28.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.278+0000 7fdab8fcf640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:16:28.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.278+0000 7fdab8fcf640 10 monclient: dump: 2024-03-29T11:16:28.268 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:16:28.268 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:28.268 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:28.268 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:28.268 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:28.268 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:28.268 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:16:28.268 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:16:28.268 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:16:28.268 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:28.268 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.278+0000 7fdab8fcf640 1 -- 172.21.15.18:0/1042874451 <== mon.1 v1:172.21.15.70:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdab0001710 con 0x7fdab413f650 2024-03-29T11:16:28.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.278+0000 7fdab8fcf640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:28.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.278+0000 7fdab8fcf640 1 -- 172.21.15.18:0/1042874451 <== mon.1 v1:172.21.15.70:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fdab0001b60 con 0x7fdab413f650 2024-03-29T11:16:28.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.278+0000 7fdab8fcf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:28.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.278+0000 7fdab8fcf640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:16:28.269 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.278+0000 7fdab8fcf640 10 monclient: dump: 2024-03-29T11:16:28.269 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:16:28.269 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:28.269 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:28.269 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:28.269 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:28.269 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:28.269 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:16:28.269 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:16:28.269 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:16:28.269 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:28.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.278+0000 7fdabc235640 5 monclient: authenticate success, global_id 4441 2024-03-29T11:16:28.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.278+0000 7fdabc235640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:28.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.278+0000 7fdabc235640 4 set_mon_vals no callback set 2024-03-29T11:16:28.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.279+0000 7fdabc235640 10 monclient: shutdown 2024-03-29T11:16:28.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.279+0000 7fdabc235640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:28.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.279+0000 7fdabc235640 1 -- 172.21.15.18:0/1042874451 >> v1:172.21.15.70:6789/0 conn(0x7fdab413f650 legacy=0x7fdab40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:28.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.279+0000 7fdabc235640 1 -- 172.21.15.18:0/1042874451 shutdown_connections 2024-03-29T11:16:28.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.279+0000 7fdabc235640 1 -- 172.21.15.18:0/1042874451 >> 172.21.15.18:0/1042874451 conn(0x7fdab4066a00 msgr2=0x7fdab4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:28.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.279+0000 7fdabc235640 1 -- 172.21.15.18:0/1042874451 shutdown_connections 2024-03-29T11:16:28.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.280+0000 7fdabc235640 1 -- 172.21.15.18:0/1042874451 wait complete. 2024-03-29T11:16:28.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.280+0000 7fdabc235640 10 monclient: build_initial_monmap 2024-03-29T11:16:28.270 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.280+0000 7fdabc235640 10 monclient: monmap: 2024-03-29T11:16:28.270 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:16:28.270 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:28.271 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:16:28.271 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:16:28.271 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:28.271 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:28.271 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:16:28.271 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:16:28.271 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:16:28.271 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:28.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.281+0000 7fdabc235640 1 librados: starting msgr at 2024-03-29T11:16:28.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.281+0000 7fdabc235640 1 librados: starting objecter 2024-03-29T11:16:28.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.281+0000 7fdabc235640 1 Processor -- start 2024-03-29T11:16:28.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.281+0000 7fdabc235640 1 -- start start 2024-03-29T11:16:28.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.281+0000 7fdabc235640 1 librados: setting wanted keys 2024-03-29T11:16:28.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.281+0000 7fdabc235640 1 librados: calling monclient init 2024-03-29T11:16:28.271 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.281+0000 7fdabc235640 10 monclient: init 2024-03-29T11:16:28.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.282+0000 7fdabc235640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:28.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.282+0000 7fdabc235640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:16:28.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.282+0000 7fdabc235640 10 monclient(hunting): picked mon.noname-b con 0x7fdab40a9d70 addr v1:172.21.15.70:6789/0 2024-03-29T11:16:28.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.282+0000 7fdabc235640 10 monclient(hunting): picked mon.noname-a con 0x7fdab413f650 addr v1:172.21.15.18:6789/0 2024-03-29T11:16:28.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.282+0000 7fdabc235640 10 monclient(hunting): picked mon.noname-c con 0x7fdab41457f0 addr v1:172.21.15.18:6790/0 2024-03-29T11:16:28.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.282+0000 7fdabc235640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab4142d00 con 0x7fdab413f650 2024-03-29T11:16:28.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.282+0000 7fdabc235640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab413aae0 con 0x7fdab40a9d70 2024-03-29T11:16:28.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.282+0000 7fdabc235640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab413adc0 con 0x7fdab41457f0 2024-03-29T11:16:28.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.282+0000 7fdabc235640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:28.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.282+0000 7fdabc235640 10 monclient(hunting): authenticate will time out at 1214.554332s 2024-03-29T11:16:28.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.282+0000 7fdab97d0640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fdab413f650 0x7fdab4140390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:47654/0 (socket says 172.21.15.18:47654) 2024-03-29T11:16:28.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.282+0000 7fdab97d0640 1 -- 172.21.15.18:0/4248471885 learned_addr learned my addr 172.21.15.18:0/4248471885 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:16:28.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.282+0000 7fdaaa7fc640 10 client.?.objecter ms_handle_connect 0x7fdab413f650 2024-03-29T11:16:28.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.282+0000 7fdaaa7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:28.272 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.282+0000 7fdaaa7fc640 10 client.?.objecter ms_handle_connect 0x7fdab41457f0 2024-03-29T11:16:28.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.282+0000 7fdaaa7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:28.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.283+0000 7fdaaa7fc640 1 -- 172.21.15.18:0/4248471885 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3085044446 0 0) 0x7fdab413adc0 con 0x7fdab41457f0 2024-03-29T11:16:28.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.283+0000 7fdaaa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:16:28.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.283+0000 7fdaaa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:28.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.283+0000 7fdaaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:28.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.283+0000 7fdaaa7fc640 10 monclient(hunting): my global_id is 4535 2024-03-29T11:16:28.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.283+0000 7fdaaa7fc640 1 -- 172.21.15.18:0/4248471885 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda98004eb0 con 0x7fdab41457f0 2024-03-29T11:16:28.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.283+0000 7fdaaa7fc640 1 -- 172.21.15.18:0/4248471885 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2920295186 0 0) 0x7fdab4142d00 con 0x7fdab413f650 2024-03-29T11:16:28.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.283+0000 7fdaaa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:16:28.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.283+0000 7fdaaa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:28.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.283+0000 7fdaaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:28.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.283+0000 7fdaaa7fc640 10 monclient(hunting): my global_id is 4473 2024-03-29T11:16:28.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.283+0000 7fdaaa7fc640 1 -- 172.21.15.18:0/4248471885 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdab413adc0 con 0x7fdab413f650 2024-03-29T11:16:28.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.283+0000 7fdaaa7fc640 10 client.?.objecter ms_handle_connect 0x7fdab40a9d70 2024-03-29T11:16:28.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.283+0000 7fdaaa7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:28.273 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.283+0000 7fdaaa7fc640 1 -- 172.21.15.18:0/4248471885 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 639748742 0 0) 0x7fdab413aae0 con 0x7fdab40a9d70 2024-03-29T11:16:28.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.283+0000 7fdaaa7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:16:28.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.283+0000 7fdaaa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:28.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.283+0000 7fdaaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:28.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.283+0000 7fdaaa7fc640 10 monclient(hunting): my global_id is 4444 2024-03-29T11:16:28.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.283+0000 7fdaaa7fc640 1 -- 172.21.15.18:0/4248471885 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdab4142d00 con 0x7fdab40a9d70 2024-03-29T11:16:28.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.283+0000 7fdaaa7fc640 1 -- 172.21.15.18:0/4248471885 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1418207969 0 0) 0x7fda98004eb0 con 0x7fdab41457f0 2024-03-29T11:16:28.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.283+0000 7fdaaa7fc640 1 -- 172.21.15.18:0/4248471885 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdab413aae0 con 0x7fdab41457f0 2024-03-29T11:16:28.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.283+0000 7fdaaa7fc640 1 -- 172.21.15.18:0/4248471885 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fdaa4001930 con 0x7fdab41457f0 2024-03-29T11:16:28.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.284+0000 7fdaaa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:28.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.284+0000 7fdaaa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:16:28.274 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.284+0000 7fdaaa7fc640 10 monclient(hunting): dump: 2024-03-29T11:16:28.274 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:16:28.274 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:28.274 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:28.275 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:28.275 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:28.275 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:28.275 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:16:28.275 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:16:28.275 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:16:28.275 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:28.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.284+0000 7fdaaa7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:28.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.284+0000 7fdaaa7fc640 1 -- 172.21.15.18:0/4248471885 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1030455179 0 0) 0x7fdab4142d00 con 0x7fdab40a9d70 2024-03-29T11:16:28.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.284+0000 7fdaaa7fc640 1 -- 172.21.15.18:0/4248471885 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fda98004eb0 con 0x7fdab40a9d70 2024-03-29T11:16:28.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.284+0000 7fdaaa7fc640 1 -- 172.21.15.18:0/4248471885 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fdaac001a50 con 0x7fdab40a9d70 2024-03-29T11:16:28.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.284+0000 7fdaaa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:28.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.284+0000 7fdaaa7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:16:28.275 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.284+0000 7fdaaa7fc640 10 monclient(hunting): dump: 2024-03-29T11:16:28.275 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:16:28.275 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:28.276 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:28.276 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:28.276 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:28.276 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:28.276 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:16:28.276 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:16:28.276 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:16:28.276 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:28.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.284+0000 7fdaaa7fc640 1 -- 172.21.15.18:0/4248471885 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 112410974 0 0) 0x7fdab413adc0 con 0x7fdab413f650 2024-03-29T11:16:28.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.284+0000 7fdaaa7fc640 1 -- 172.21.15.18:0/4248471885 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdab4142d00 con 0x7fdab413f650 2024-03-29T11:16:28.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.284+0000 7fdaaa7fc640 1 -- 172.21.15.18:0/4248471885 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fdab0001ae0 con 0x7fdab413f650 2024-03-29T11:16:28.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.284+0000 7fdaaa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:28.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.284+0000 7fdaaa7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:16:28.276 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.284+0000 7fdaaa7fc640 10 monclient(hunting): dump: 2024-03-29T11:16:28.276 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:16:28.276 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:28.277 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:28.277 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:28.277 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:28.277 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:28.277 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:16:28.277 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:16:28.277 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:16:28.277 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:28.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.284+0000 7fdaaa7fc640 1 -- 172.21.15.18:0/4248471885 <== mon.1 v1:172.21.15.70:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 604758525 0 0) 0x7fda98004eb0 con 0x7fdab40a9d70 2024-03-29T11:16:28.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.284+0000 7fdaaa7fc640 1 -- 172.21.15.18:0/4248471885 >> v1:172.21.15.18:6790/0 conn(0x7fdab41457f0 legacy=0x7fdab4140aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:28.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.284+0000 7fdaaa7fc640 1 -- 172.21.15.18:0/4248471885 >> v1:172.21.15.18:6789/0 conn(0x7fdab413f650 legacy=0x7fdab4140390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:28.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.284+0000 7fdaaa7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:16:28.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.284+0000 7fdaaa7fc640 1 monclient: found mon.b 2024-03-29T11:16:28.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.284+0000 7fdaaa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:28.277 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.284+0000 7fdaaa7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:16:28.278 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.284+0000 7fdaaa7fc640 1 -- 172.21.15.18:0/4248471885 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab413b0a0 con 0x7fdab40a9d70 2024-03-29T11:16:28.278 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.284+0000 7fdaaa7fc640 10 monclient: _finish_auth 0 2024-03-29T11:16:28.278 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.284+0000 7fdaaa7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:16:28.278 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.284+0000 7fdaaa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:28.278 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.284+0000 7fdabc235640 5 monclient: authenticate success, global_id 4444 2024-03-29T11:16:28.278 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.284+0000 7fdabc235640 10 monclient: _renew_subs 2024-03-29T11:16:28.278 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.284+0000 7fdabc235640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:16:28.278 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.284+0000 7fdabc235640 1 -- 172.21.15.18:0/4248471885 --> v1:172.21.15.70:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdab4071d30 con 0x7fdab40a9d70 2024-03-29T11:16:28.278 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.284+0000 7fdabc235640 10 client.4444.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:28.278 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.284+0000 7fdabc235640 10 monclient: _renew_subs 2024-03-29T11:16:28.278 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.284+0000 7fdabc235640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:16:28.278 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.284+0000 7fdabc235640 1 -- 172.21.15.18:0/4248471885 --> v1:172.21.15.70:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdab4139f20 con 0x7fdab40a9d70 2024-03-29T11:16:28.278 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.284+0000 7fdabc235640 1 librados: init done 2024-03-29T11:16:28.278 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.285+0000 7fdaaa7fc640 1 -- 172.21.15.18:0/4248471885 <== mon.1 v1:172.21.15.70:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdaac001f60 con 0x7fdab40a9d70 2024-03-29T11:16:28.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.285+0000 7fdaaa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:28.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.285+0000 7fdaaa7fc640 1 -- 172.21.15.18:0/4248471885 <== mon.1 v1:172.21.15.70:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fdaac004d00 con 0x7fdab40a9d70 2024-03-29T11:16:28.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.285+0000 7fdaaa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:28.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.285+0000 7fdaaa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:16:28.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.285+0000 7fdaaa7fc640 10 monclient: dump: 2024-03-29T11:16:28.279 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:16:28.279 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:28.279 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:28.279 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:28.279 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:28.279 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:28.279 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:16:28.279 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:16:28.279 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:16:28.279 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:28.279 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.285+0000 7fdaabfff640 4 set_mon_vals no callback set 2024-03-29T11:16:28.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.286+0000 7fdabc235640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:28.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.286+0000 7fdabc235640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:28.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.286+0000 7fdabc235640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:16:28.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.286+0000 7fdabc235640 1 -- 172.21.15.18:0/4248471885 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda9c005080 con 0x7fdab40a9d70 2024-03-29T11:16:28.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.287+0000 7fdaaa7fc640 1 -- 172.21.15.18:0/4248471885 <== mon.1 v1:172.21.15.70:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7fdaac01acd0 con 0x7fdab40a9d70 2024-03-29T11:16:28.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.287+0000 7fdaaa7fc640 1 -- 172.21.15.18:0/4248471885 <== mon.1 v1:172.21.15.70:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7fdaac084c20 con 0x7fdab40a9d70 2024-03-29T11:16:28.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.287+0000 7fdaaa7fc640 10 client.4444.objecter ms_dispatch 0x7fdab4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:28.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.287+0000 7fdaaa7fc640 3 client.4444.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:28.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.287+0000 7fdaaa7fc640 3 client.4444.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:28.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.288+0000 7fdaaa7fc640 20 client.4444.objecter dump_active .. 0 homeless 2024-03-29T11:16:28.280 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.288+0000 7fdab97d0640 10 monclient: get_auth_request con 0x7fda9806e150 auth_method 0 2024-03-29T11:16:28.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.289+0000 7fdaaa7fc640 10 client.4444.objecter ms_handle_connect 0x7fda9806e150 2024-03-29T11:16:28.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.297+0000 7fdaaa7fc640 1 -- 172.21.15.18:0/4248471885 <== mon.1 v1:172.21.15.70:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdaac050500 con 0x7fdab40a9d70 2024-03-29T11:16:28.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.297+0000 7fdaaa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:28.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.297+0000 7fdaaa7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:28.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.687+0000 7fdabc235640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:28.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.687+0000 7fdabc235640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:28.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.687+0000 7fdabc235640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:16:28.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.687+0000 7fdabc235640 1 -- 172.21.15.18:0/4248471885 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda9c005330 con 0x7fdab40a9d70 2024-03-29T11:16:28.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.690+0000 7fdaaa7fc640 1 -- 172.21.15.18:0/4248471885 <== mon.1 v1:172.21.15.70:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7fdaac052430 con 0x7fdab40a9d70 2024-03-29T11:16:28.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.690+0000 7fdaaa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:28.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.690+0000 7fdaaa7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:28.667 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:16:28.667 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:28.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.693+0000 7fdabc235640 10 librados: watch_flush enter 2024-03-29T11:16:28.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.693+0000 7fdabc235640 10 librados: watch_flush exit 2024-03-29T11:16:28.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.693+0000 7fdabc235640 20 client.4444.objecter shutdown clearing up homeless session... 2024-03-29T11:16:28.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.693+0000 7fdabc235640 10 client.4444.objecter successfully canceled tick 2024-03-29T11:16:28.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.693+0000 7fdabc235640 1 -- 172.21.15.18:0/4248471885 >> v1:172.21.15.70:6816/3101871562 conn(0x7fda9806e150 legacy=0x7fda98070610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:28.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.693+0000 7fdabc235640 10 monclient: shutdown 2024-03-29T11:16:28.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.693+0000 7fdabc235640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:28.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.693+0000 7fdabc235640 1 -- 172.21.15.18:0/4248471885 >> v1:172.21.15.70:6789/0 conn(0x7fdab40a9d70 legacy=0x7fdab413b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:28.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.693+0000 7fdabc235640 1 -- 172.21.15.18:0/4248471885 shutdown_connections 2024-03-29T11:16:28.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.693+0000 7fdabc235640 1 -- 172.21.15.18:0/4248471885 >> 172.21.15.18:0/4248471885 conn(0x7fdab4066a00 msgr2=0x7fdab40b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:28.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.693+0000 7fdabc235640 1 -- 172.21.15.18:0/4248471885 shutdown_connections 2024-03-29T11:16:28.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.694+0000 7fdabc235640 1 -- 172.21.15.18:0/4248471885 wait complete. 2024-03-29T11:16:28.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:28.694+0000 7fdabc235640 1 librados: shutdown 2024-03-29T11:16:28.687 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:35.493 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:35.647 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.667+0000 7f27eb3cf640 10 monclient: get_monmap_and_config 2024-03-29T11:16:35.647 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.667+0000 7f27eb3cf640 10 monclient: build_initial_monmap 2024-03-29T11:16:35.647 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.667+0000 7f27eb3cf640 10 monclient: monmap: 2024-03-29T11:16:35.647 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:16:35.647 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:35.648 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:16:35.668289+0000 2024-03-29T11:16:35.648 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:16:35.668289+0000 2024-03-29T11:16:35.648 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:35.648 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:35.648 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:16:35.648 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:16:35.648 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:16:35.648 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:35.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.668+0000 7f27eb3cf640 1 Processor -- start 2024-03-29T11:16:35.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.668+0000 7f27eb3cf640 1 -- start start 2024-03-29T11:16:35.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.668+0000 7f27eb3cf640 10 monclient: init 2024-03-29T11:16:35.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.669+0000 7f27eb3cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:35.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.669+0000 7f27eb3cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:16:35.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.669+0000 7f27eb3cf640 10 monclient(hunting): picked mon.noname-c con 0x7f27e40ad6c0 addr v1:172.21.15.70:6789/0 2024-03-29T11:16:35.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.669+0000 7f27eb3cf640 10 monclient(hunting): picked mon.noname-a con 0x7f27e4152980 addr v1:172.21.15.18:6789/0 2024-03-29T11:16:35.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.669+0000 7f27eb3cf640 10 monclient(hunting): picked mon.noname-b con 0x7f27e41588d0 addr v1:172.21.15.18:6790/0 2024-03-29T11:16:35.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.669+0000 7f27eb3cf640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27e40a94e0 con 0x7f27e4152980 2024-03-29T11:16:35.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.669+0000 7f27eb3cf640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27e40af3a0 con 0x7f27e40ad6c0 2024-03-29T11:16:35.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.669+0000 7f27eb3cf640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27e40afb60 con 0x7f27e41588d0 2024-03-29T11:16:35.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.669+0000 7f27eb3cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:35.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.669+0000 7f27eb3cf640 10 monclient(hunting): authenticate will time out at 1221.941467s 2024-03-29T11:16:35.650 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.671+0000 7f27e996c640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f27e41588d0 0x7f27e415acc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:40208/0 (socket says 172.21.15.18:40208) 2024-03-29T11:16:35.650 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.671+0000 7f27e996c640 1 -- 172.21.15.18:0/3618632517 learned_addr learned my addr 172.21.15.18:0/3618632517 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:16:35.650 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.671+0000 7f27dbfff640 1 -- 172.21.15.18:0/3618632517 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2581568496 0 0) 0x7f27e40afb60 con 0x7f27e41588d0 2024-03-29T11:16:35.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.671+0000 7f27dbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:16:35.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.671+0000 7f27dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:35.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.671+0000 7f27dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:35.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.671+0000 7f27dbfff640 10 monclient(hunting): my global_id is 4538 2024-03-29T11:16:35.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.671+0000 7f27dbfff640 1 -- 172.21.15.18:0/3618632517 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27d0003520 con 0x7f27e41588d0 2024-03-29T11:16:35.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.671+0000 7f27dbfff640 1 -- 172.21.15.18:0/3618632517 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2339545733 0 0) 0x7f27d0003520 con 0x7f27e41588d0 2024-03-29T11:16:35.651 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.671+0000 7f27dbfff640 1 -- 172.21.15.18:0/3618632517 >> v1:172.21.15.70:6789/0 conn(0x7f27e40ad6c0 legacy=0x7f27e414fb00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:16:35.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.675+0000 7f27dbfff640 1 -- 172.21.15.18:0/3618632517 >> v1:172.21.15.18:6789/0 conn(0x7f27e4152980 legacy=0x7f27e4152df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:16:35.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.675+0000 7f27dbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:16:35.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.675+0000 7f27dbfff640 1 monclient: found mon.noname-b 2024-03-29T11:16:35.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.675+0000 7f27dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:35.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.675+0000 7f27dbfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:16:35.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.675+0000 7f27dbfff640 1 -- 172.21.15.18:0/3618632517 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e40aab80 con 0x7f27e41588d0 2024-03-29T11:16:35.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.675+0000 7f27dbfff640 10 monclient: _finish_auth 0 2024-03-29T11:16:35.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.675+0000 7f27dbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:16:35.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.675+0000 7f27dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:35.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.675+0000 7f27dbfff640 1 -- 172.21.15.18:0/3618632517 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f27dc0019f0 con 0x7f27e41588d0 2024-03-29T11:16:35.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.675+0000 7f27dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:35.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.675+0000 7f27dbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:16:35.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.675+0000 7f27dbfff640 10 monclient: dump: 2024-03-29T11:16:35.655 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:16:35.655 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:35.655 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:35.655 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:35.655 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:35.655 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:35.655 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:16:35.655 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:16:35.655 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:16:35.656 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:35.656 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.676+0000 7f27eb3cf640 5 monclient: authenticate success, global_id 4538 2024-03-29T11:16:35.656 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.676+0000 7f27eb3cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:35.656 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.676+0000 7f27dbfff640 1 -- 172.21.15.18:0/3618632517 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27dc001c50 con 0x7f27e41588d0 2024-03-29T11:16:35.656 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.676+0000 7f27dbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:35.656 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.676+0000 7f27dbfff640 1 -- 172.21.15.18:0/3618632517 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f27dc0020a0 con 0x7f27e41588d0 2024-03-29T11:16:35.656 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.676+0000 7f27dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:35.656 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.676+0000 7f27dbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:35.656 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.676+0000 7f27dbfff640 10 monclient: dump: 2024-03-29T11:16:35.656 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:16:35.656 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:35.656 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:35.656 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:35.657 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:35.657 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:35.657 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:16:35.657 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:16:35.657 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:16:35.657 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:35.658 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.677+0000 7f27eb3cf640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:35.658 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.677+0000 7f27eb3cf640 4 set_mon_vals no callback set 2024-03-29T11:16:35.658 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.678+0000 7f27eb3cf640 10 monclient: shutdown 2024-03-29T11:16:35.658 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.678+0000 7f27eb3cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:35.658 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.678+0000 7f27eb3cf640 1 -- 172.21.15.18:0/3618632517 >> v1:172.21.15.18:6790/0 conn(0x7f27e41588d0 legacy=0x7f27e415acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:35.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.679+0000 7f27eb3cf640 1 -- 172.21.15.18:0/3618632517 shutdown_connections 2024-03-29T11:16:35.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.679+0000 7f27eb3cf640 1 -- 172.21.15.18:0/3618632517 >> 172.21.15.18:0/3618632517 conn(0x7f27e40b35e0 msgr2=0x7f27e40b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:35.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.680+0000 7f27eb3cf640 1 -- 172.21.15.18:0/3618632517 shutdown_connections 2024-03-29T11:16:35.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.682+0000 7f27eb3cf640 1 -- 172.21.15.18:0/3618632517 wait complete. 2024-03-29T11:16:35.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.683+0000 7f27eb3cf640 10 monclient: build_initial_monmap 2024-03-29T11:16:35.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.683+0000 7f27eb3cf640 10 monclient: monmap: 2024-03-29T11:16:35.662 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:16:35.662 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:35.662 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:16:35.662 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:16:35.663 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:35.663 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:35.663 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:16:35.663 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:16:35.663 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:16:35.663 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:35.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.683+0000 7f27eb3cf640 1 librados: starting msgr at 2024-03-29T11:16:35.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.683+0000 7f27eb3cf640 1 librados: starting objecter 2024-03-29T11:16:35.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.683+0000 7f27eb3cf640 1 Processor -- start 2024-03-29T11:16:35.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.684+0000 7f27eb3cf640 1 -- start start 2024-03-29T11:16:35.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.684+0000 7f27eb3cf640 1 librados: setting wanted keys 2024-03-29T11:16:35.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.684+0000 7f27eb3cf640 1 librados: calling monclient init 2024-03-29T11:16:35.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.684+0000 7f27eb3cf640 10 monclient: init 2024-03-29T11:16:35.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.685+0000 7f27eb3cf640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:35.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.685+0000 7f27eb3cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:16:35.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.685+0000 7f27eb3cf640 10 monclient(hunting): picked mon.noname-c con 0x7f27e40ad6c0 addr v1:172.21.15.18:6790/0 2024-03-29T11:16:35.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.685+0000 7f27e916b640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f27e40ad6c0 0x7f27e414c7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:49442/0 (socket says 172.21.15.18:49442) 2024-03-29T11:16:35.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.686+0000 7f27e916b640 1 -- 172.21.15.18:0/3535602234 learned_addr learned my addr 172.21.15.18:0/3535602234 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:16:35.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.686+0000 7f27eb3cf640 10 monclient(hunting): picked mon.noname-a con 0x7f27e4152980 addr v1:172.21.15.18:6789/0 2024-03-29T11:16:35.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.686+0000 7f27eb3cf640 10 monclient(hunting): picked mon.noname-b con 0x7f27e41588d0 addr v1:172.21.15.70:6789/0 2024-03-29T11:16:35.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.686+0000 7f27eb3cf640 1 -- 172.21.15.18:0/3535602234 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27e4071f40 con 0x7f27e4152980 2024-03-29T11:16:35.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.686+0000 7f27eb3cf640 1 -- 172.21.15.18:0/3535602234 --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27e406f850 con 0x7f27e41588d0 2024-03-29T11:16:35.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.686+0000 7f27eb3cf640 1 -- 172.21.15.18:0/3535602234 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27e406fb30 con 0x7f27e40ad6c0 2024-03-29T11:16:35.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.686+0000 7f27eb3cf640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:35.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.686+0000 7f27eb3cf640 10 monclient(hunting): authenticate will time out at 1221.958546s 2024-03-29T11:16:35.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.686+0000 7f27d9ffb640 10 client.?.objecter ms_handle_connect 0x7f27e40ad6c0 2024-03-29T11:16:35.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.686+0000 7f27d9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:35.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.686+0000 7f27d9ffb640 1 -- 172.21.15.18:0/3535602234 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3865411443 0 0) 0x7f27e406fb30 con 0x7f27e40ad6c0 2024-03-29T11:16:35.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.686+0000 7f27d9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:16:35.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.686+0000 7f27d9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:35.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.686+0000 7f27d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:35.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.687+0000 7f27d9ffb640 10 monclient(hunting): my global_id is 4541 2024-03-29T11:16:35.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.687+0000 7f27d9ffb640 1 -- 172.21.15.18:0/3535602234 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27c8004eb0 con 0x7f27e40ad6c0 2024-03-29T11:16:35.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.687+0000 7f27d9ffb640 1 -- 172.21.15.18:0/3535602234 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3168091438 0 0) 0x7f27c8004eb0 con 0x7f27e40ad6c0 2024-03-29T11:16:35.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.687+0000 7f27d9ffb640 1 -- 172.21.15.18:0/3535602234 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27e406fb30 con 0x7f27e40ad6c0 2024-03-29T11:16:35.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.687+0000 7f27d9ffb640 1 -- 172.21.15.18:0/3535602234 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f27d4001870 con 0x7f27e40ad6c0 2024-03-29T11:16:35.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.687+0000 7f27d9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:35.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.687+0000 7f27d9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:16:35.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.687+0000 7f27d9ffb640 10 monclient(hunting): dump: 2024-03-29T11:16:35.668 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:16:35.668 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:35.668 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:35.668 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:35.668 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:35.669 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:35.669 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:16:35.669 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:16:35.669 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:16:35.669 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:35.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.688+0000 7f27d9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:35.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.688+0000 7f27d9ffb640 1 -- 172.21.15.18:0/3535602234 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2254762830 0 0) 0x7f27e406fb30 con 0x7f27e40ad6c0 2024-03-29T11:16:35.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.688+0000 7f27d9ffb640 1 -- 172.21.15.18:0/3535602234 >> v1:172.21.15.70:6789/0 conn(0x7f27e41588d0 legacy=0x7f27e4070760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:16:35.669 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.688+0000 7f27d9ffb640 1 -- 172.21.15.18:0/3535602234 >> v1:172.21.15.18:6789/0 conn(0x7f27e4152980 legacy=0x7f27e4070050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:16:35.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.691+0000 7f27d9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:16:35.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.691+0000 7f27d9ffb640 1 monclient: found mon.c 2024-03-29T11:16:35.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.691+0000 7f27d9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:35.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.692+0000 7f27d9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:16:35.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.692+0000 7f27d9ffb640 1 -- 172.21.15.18:0/3535602234 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27e41e4e90 con 0x7f27e40ad6c0 2024-03-29T11:16:35.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.692+0000 7f27d9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:16:35.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.692+0000 7f27d9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:16:35.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.692+0000 7f27d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:35.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.692+0000 7f27eb3cf640 5 monclient: authenticate success, global_id 4541 2024-03-29T11:16:35.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.692+0000 7f27eb3cf640 10 monclient: _renew_subs 2024-03-29T11:16:35.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.692+0000 7f27eb3cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:16:35.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.692+0000 7f27eb3cf640 1 -- 172.21.15.18:0/3535602234 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27e41e5140 con 0x7f27e40ad6c0 2024-03-29T11:16:35.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.692+0000 7f27eb3cf640 10 client.4541.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:35.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.693+0000 7f27eb3cf640 10 monclient: _renew_subs 2024-03-29T11:16:35.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.693+0000 7f27eb3cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:16:35.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.693+0000 7f27eb3cf640 1 -- 172.21.15.18:0/3535602234 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f27e414d090 con 0x7f27e40ad6c0 2024-03-29T11:16:35.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.693+0000 7f27eb3cf640 1 librados: init done 2024-03-29T11:16:35.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.692+0000 7f27d9ffb640 1 -- 172.21.15.18:0/3535602234 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27d40019d0 con 0x7f27e40ad6c0 2024-03-29T11:16:35.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.693+0000 7f27d9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:35.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.693+0000 7f27d9ffb640 1 -- 172.21.15.18:0/3535602234 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f27d4004ae0 con 0x7f27e40ad6c0 2024-03-29T11:16:35.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.693+0000 7f27d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:35.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.693+0000 7f27d9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:35.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.693+0000 7f27d9ffb640 10 monclient: dump: 2024-03-29T11:16:35.673 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:16:35.673 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:35.673 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:35.674 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:35.674 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:35.674 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:35.674 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:16:35.674 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:16:35.674 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:16:35.674 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:35.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.693+0000 7f27db7fe640 4 set_mon_vals no callback set 2024-03-29T11:16:35.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.696+0000 7f27eb3cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:35.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.696+0000 7f27eb3cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:35.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.696+0000 7f27eb3cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:16:35.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.696+0000 7f27eb3cf640 1 -- 172.21.15.18:0/3535602234 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27c4005080 con 0x7f27e40ad6c0 2024-03-29T11:16:35.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.697+0000 7f27d9ffb640 1 -- 172.21.15.18:0/3535602234 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f27d401aab0 con 0x7f27e40ad6c0 2024-03-29T11:16:35.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.698+0000 7f27d9ffb640 1 -- 172.21.15.18:0/3535602234 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f27d4084b50 con 0x7f27e40ad6c0 2024-03-29T11:16:35.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.698+0000 7f27d9ffb640 10 client.4541.objecter ms_dispatch 0x7f27e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:35.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.698+0000 7f27d9ffb640 3 client.4541.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:35.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.698+0000 7f27d9ffb640 3 client.4541.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:35.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.699+0000 7f27d9ffb640 20 client.4541.objecter dump_active .. 0 homeless 2024-03-29T11:16:35.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.700+0000 7f27e896a640 10 monclient: get_auth_request con 0x7f27c806dcb0 auth_method 0 2024-03-29T11:16:35.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.702+0000 7f27d9ffb640 10 client.4541.objecter ms_handle_connect 0x7f27c806dcb0 2024-03-29T11:16:35.684 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.707+0000 7f27d9ffb640 1 -- 172.21.15.18:0/3535602234 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f27d4050260 con 0x7f27e40ad6c0 2024-03-29T11:16:35.684 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.707+0000 7f27d9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:35.684 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:35.707+0000 7f27d9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:36.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:36.044+0000 7f27eb3cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:36.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:36.045+0000 7f27eb3cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:36.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:36.045+0000 7f27eb3cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:16:36.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:36.045+0000 7f27eb3cf640 1 -- 172.21.15.18:0/3535602234 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27c4005330 con 0x7f27e40ad6c0 2024-03-29T11:16:36.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:36.046+0000 7f27d9ffb640 1 -- 172.21.15.18:0/3535602234 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f27d4052360 con 0x7f27e40ad6c0 2024-03-29T11:16:36.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:36.046+0000 7f27d9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:36.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:36.046+0000 7f27d9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:36.026 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:16:36.026 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:36.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:36.051+0000 7f27eb3cf640 10 librados: watch_flush enter 2024-03-29T11:16:36.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:36.051+0000 7f27eb3cf640 10 librados: watch_flush exit 2024-03-29T11:16:36.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:36.051+0000 7f27eb3cf640 20 client.4541.objecter shutdown clearing up homeless session... 2024-03-29T11:16:36.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:36.051+0000 7f27eb3cf640 10 client.4541.objecter successfully canceled tick 2024-03-29T11:16:36.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:36.052+0000 7f27eb3cf640 1 -- 172.21.15.18:0/3535602234 >> v1:172.21.15.70:6816/3101871562 conn(0x7f27c806dcb0 legacy=0x7f27c8070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:36.030 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:36.052+0000 7f27eb3cf640 10 monclient: shutdown 2024-03-29T11:16:36.030 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:36.052+0000 7f27eb3cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:36.030 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:36.052+0000 7f27eb3cf640 1 -- 172.21.15.18:0/3535602234 >> v1:172.21.15.18:6790/0 conn(0x7f27e40ad6c0 legacy=0x7f27e414c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:36.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:36.057+0000 7f27eb3cf640 1 -- 172.21.15.18:0/3535602234 shutdown_connections 2024-03-29T11:16:36.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:36.057+0000 7f27eb3cf640 1 -- 172.21.15.18:0/3535602234 >> 172.21.15.18:0/3535602234 conn(0x7f27e40b35e0 msgr2=0x7f27e40b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:36.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:36.057+0000 7f27eb3cf640 1 -- 172.21.15.18:0/3535602234 shutdown_connections 2024-03-29T11:16:36.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:36.057+0000 7f27eb3cf640 1 -- 172.21.15.18:0/3535602234 wait complete. 2024-03-29T11:16:36.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:36.057+0000 7f27eb3cf640 1 librados: shutdown 2024-03-29T11:16:36.055 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:42.861 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:43.044 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.066+0000 7fa9da243640 10 monclient: get_monmap_and_config 2024-03-29T11:16:43.044 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.066+0000 7fa9da243640 10 monclient: build_initial_monmap 2024-03-29T11:16:43.044 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.066+0000 7fa9da243640 10 monclient: monmap: 2024-03-29T11:16:43.044 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:16:43.044 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:43.045 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:16:43.068059+0000 2024-03-29T11:16:43.045 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:16:43.068059+0000 2024-03-29T11:16:43.045 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:43.045 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:43.045 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:16:43.045 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:16:43.045 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:16:43.045 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:43.045 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.067+0000 7fa9da243640 1 Processor -- start 2024-03-29T11:16:43.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.067+0000 7fa9da243640 1 -- start start 2024-03-29T11:16:43.046 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.067+0000 7fa9da243640 10 monclient: init 2024-03-29T11:16:43.047 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.068+0000 7fa9da243640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:43.047 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.068+0000 7fa9da243640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:16:43.047 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.068+0000 7fa9da243640 10 monclient(hunting): picked mon.noname-a con 0x7fa9d41457f0 addr v1:172.21.15.18:6789/0 2024-03-29T11:16:43.047 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.068+0000 7fa9da243640 10 monclient(hunting): picked mon.noname-b con 0x7fa9d413f650 addr v1:172.21.15.18:6790/0 2024-03-29T11:16:43.047 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.068+0000 7fa9da243640 10 monclient(hunting): picked mon.noname-c con 0x7fa9d40a9d70 addr v1:172.21.15.70:6789/0 2024-03-29T11:16:43.047 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.068+0000 7fa9da243640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d40aaef0 con 0x7fa9d41457f0 2024-03-29T11:16:43.047 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.068+0000 7fa9da243640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d4145bd0 con 0x7fa9d40a9d70 2024-03-29T11:16:43.047 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.068+0000 7fa9da243640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d4145eb0 con 0x7fa9d413f650 2024-03-29T11:16:43.047 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.068+0000 7fa9da243640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:43.047 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.068+0000 7fa9da243640 10 monclient(hunting): authenticate will time out at 1229.341009s 2024-03-29T11:16:43.048 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.069+0000 7fa9d2ffd640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fa9d413f650 0x7fa9d40b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:38996/0 (socket says 172.21.15.18:38996) 2024-03-29T11:16:43.048 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.069+0000 7fa9d2ffd640 1 -- 172.21.15.18:0/503906687 learned_addr learned my addr 172.21.15.18:0/503906687 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:16:43.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.069+0000 7fa9d27fc640 1 -- 172.21.15.18:0/503906687 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4062654538 0 0) 0x7fa9d4145eb0 con 0x7fa9d413f650 2024-03-29T11:16:43.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.069+0000 7fa9d27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:16:43.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.069+0000 7fa9d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:43.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.069+0000 7fa9d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:43.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.069+0000 7fa9d27fc640 10 monclient(hunting): my global_id is 4544 2024-03-29T11:16:43.050 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.069+0000 7fa9d27fc640 1 -- 172.21.15.18:0/503906687 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9bc003520 con 0x7fa9d413f650 2024-03-29T11:16:43.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.070+0000 7fa9d27fc640 1 -- 172.21.15.18:0/503906687 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3286416726 0 0) 0x7fa9d40aaef0 con 0x7fa9d41457f0 2024-03-29T11:16:43.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.070+0000 7fa9d27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:16:43.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.070+0000 7fa9d27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:43.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.070+0000 7fa9d27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:43.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.070+0000 7fa9d27fc640 10 monclient(hunting): my global_id is 4476 2024-03-29T11:16:43.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.070+0000 7fa9d27fc640 1 -- 172.21.15.18:0/503906687 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9d4145eb0 con 0x7fa9d41457f0 2024-03-29T11:16:43.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.070+0000 7fa9d27fc640 1 -- 172.21.15.18:0/503906687 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2440113583 0 0) 0x7fa9bc003520 con 0x7fa9d413f650 2024-03-29T11:16:43.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.070+0000 7fa9d27fc640 1 -- 172.21.15.18:0/503906687 >> v1:172.21.15.70:6789/0 conn(0x7fa9d40a9d70 legacy=0x7fa9d414d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:16:43.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.071+0000 7fa9d27fc640 1 -- 172.21.15.18:0/503906687 >> v1:172.21.15.18:6789/0 conn(0x7fa9d41457f0 legacy=0x7fa9d413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:43.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.071+0000 7fa9d27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:16:43.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.071+0000 7fa9d27fc640 1 monclient: found mon.noname-b 2024-03-29T11:16:43.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.071+0000 7fa9d27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:43.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.071+0000 7fa9d27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:16:43.051 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.071+0000 7fa9d27fc640 1 -- 172.21.15.18:0/503906687 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d4146340 con 0x7fa9d413f650 2024-03-29T11:16:43.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.071+0000 7fa9d27fc640 10 monclient: _finish_auth 0 2024-03-29T11:16:43.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.071+0000 7fa9d27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:16:43.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.071+0000 7fa9d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:43.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.071+0000 7fa9d27fc640 1 -- 172.21.15.18:0/503906687 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fa9c4001690 con 0x7fa9d413f650 2024-03-29T11:16:43.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.071+0000 7fa9d27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:43.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.071+0000 7fa9d27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:16:43.052 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.071+0000 7fa9d27fc640 10 monclient: dump: 2024-03-29T11:16:43.052 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:16:43.052 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:43.052 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:43.052 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:43.052 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:43.052 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:43.052 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:16:43.052 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:16:43.053 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:16:43.053 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:43.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.071+0000 7fa9da243640 5 monclient: authenticate success, global_id 4544 2024-03-29T11:16:43.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.071+0000 7fa9da243640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:43.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.072+0000 7fa9d27fc640 1 -- 172.21.15.18:0/503906687 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9c40017f0 con 0x7fa9d413f650 2024-03-29T11:16:43.053 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.072+0000 7fa9d27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:43.056 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.072+0000 7fa9da243640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:43.056 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.072+0000 7fa9da243640 4 set_mon_vals no callback set 2024-03-29T11:16:43.056 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.073+0000 7fa9da243640 10 monclient: shutdown 2024-03-29T11:16:43.056 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.073+0000 7fa9da243640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:43.056 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.073+0000 7fa9da243640 1 -- 172.21.15.18:0/503906687 >> v1:172.21.15.18:6790/0 conn(0x7fa9d413f650 legacy=0x7fa9d40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:43.056 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.073+0000 7fa9da243640 1 -- 172.21.15.18:0/503906687 shutdown_connections 2024-03-29T11:16:43.056 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.073+0000 7fa9da243640 1 -- 172.21.15.18:0/503906687 >> 172.21.15.18:0/503906687 conn(0x7fa9d4066a00 msgr2=0x7fa9d4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:43.057 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.074+0000 7fa9da243640 1 -- 172.21.15.18:0/503906687 shutdown_connections 2024-03-29T11:16:43.057 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.074+0000 7fa9da243640 1 -- 172.21.15.18:0/503906687 wait complete. 2024-03-29T11:16:43.057 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.074+0000 7fa9da243640 10 monclient: build_initial_monmap 2024-03-29T11:16:43.057 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.074+0000 7fa9da243640 10 monclient: monmap: 2024-03-29T11:16:43.057 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:16:43.057 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:43.057 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:16:43.057 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:16:43.057 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:43.057 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:43.057 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:16:43.057 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:16:43.057 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:16:43.058 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:43.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.075+0000 7fa9da243640 1 librados: starting msgr at 2024-03-29T11:16:43.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.075+0000 7fa9da243640 1 librados: starting objecter 2024-03-29T11:16:43.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.075+0000 7fa9da243640 1 Processor -- start 2024-03-29T11:16:43.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.075+0000 7fa9da243640 1 -- start start 2024-03-29T11:16:43.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.075+0000 7fa9da243640 1 librados: setting wanted keys 2024-03-29T11:16:43.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.075+0000 7fa9da243640 1 librados: calling monclient init 2024-03-29T11:16:43.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.075+0000 7fa9da243640 10 monclient: init 2024-03-29T11:16:43.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.076+0000 7fa9da243640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:43.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.076+0000 7fa9da243640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:16:43.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.076+0000 7fa9da243640 10 monclient(hunting): picked mon.noname-c con 0x7fa9d40a9d70 addr v1:172.21.15.18:6790/0 2024-03-29T11:16:43.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.076+0000 7fa9da243640 10 monclient(hunting): picked mon.noname-a con 0x7fa9d413f650 addr v1:172.21.15.18:6789/0 2024-03-29T11:16:43.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.076+0000 7fa9da243640 10 monclient(hunting): picked mon.noname-b con 0x7fa9d41457f0 addr v1:172.21.15.70:6789/0 2024-03-29T11:16:43.058 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.076+0000 7fa9da243640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d4141510 con 0x7fa9d413f650 2024-03-29T11:16:43.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.076+0000 7fa9da243640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d41417f0 con 0x7fa9d41457f0 2024-03-29T11:16:43.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.076+0000 7fa9da243640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d4141ad0 con 0x7fa9d40a9d70 2024-03-29T11:16:43.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.076+0000 7fa9da243640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:43.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.076+0000 7fa9da243640 10 monclient(hunting): authenticate will time out at 1229.349625s 2024-03-29T11:16:43.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.078+0000 7fa9d2ffd640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fa9d413f650 0x7fa9d41421d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:40862/0 (socket says 172.21.15.18:40862) 2024-03-29T11:16:43.059 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.078+0000 7fa9d2ffd640 1 -- 172.21.15.18:0/2738488799 learned_addr learned my addr 172.21.15.18:0/2738488799 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:16:43.064 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.079+0000 7fa9bbfff640 10 client.?.objecter ms_handle_connect 0x7fa9d40a9d70 2024-03-29T11:16:43.064 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.079+0000 7fa9bbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:43.064 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.080+0000 7fa9bbfff640 10 client.?.objecter ms_handle_connect 0x7fa9d413f650 2024-03-29T11:16:43.064 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.080+0000 7fa9bbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:43.064 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.080+0000 7fa9bbfff640 10 client.?.objecter ms_handle_connect 0x7fa9d41457f0 2024-03-29T11:16:43.064 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.080+0000 7fa9bbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:43.064 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.080+0000 7fa9bbfff640 1 -- 172.21.15.18:0/2738488799 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1247438060 0 0) 0x7fa9d4141ad0 con 0x7fa9d40a9d70 2024-03-29T11:16:43.064 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.080+0000 7fa9bbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:16:43.065 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.080+0000 7fa9bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:43.065 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.080+0000 7fa9bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:43.065 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.080+0000 7fa9bbfff640 10 monclient(hunting): my global_id is 4547 2024-03-29T11:16:43.065 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.080+0000 7fa9bbfff640 1 -- 172.21.15.18:0/2738488799 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9c0004eb0 con 0x7fa9d40a9d70 2024-03-29T11:16:43.065 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.080+0000 7fa9bbfff640 1 -- 172.21.15.18:0/2738488799 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1162362729 0 0) 0x7fa9d4141510 con 0x7fa9d413f650 2024-03-29T11:16:43.066 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.080+0000 7fa9bbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:16:43.066 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.080+0000 7fa9bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:43.066 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.080+0000 7fa9bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:43.066 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.080+0000 7fa9bbfff640 10 monclient(hunting): my global_id is 4479 2024-03-29T11:16:43.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.080+0000 7fa9bbfff640 1 -- 172.21.15.18:0/2738488799 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9d4141ad0 con 0x7fa9d413f650 2024-03-29T11:16:43.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.080+0000 7fa9bbfff640 1 -- 172.21.15.18:0/2738488799 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3000181108 0 0) 0x7fa9d41417f0 con 0x7fa9d41457f0 2024-03-29T11:16:43.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.080+0000 7fa9bbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:16:43.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.080+0000 7fa9bbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:43.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.080+0000 7fa9bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:43.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.080+0000 7fa9bbfff640 10 monclient(hunting): my global_id is 4447 2024-03-29T11:16:43.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.080+0000 7fa9bbfff640 1 -- 172.21.15.18:0/2738488799 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9d4141510 con 0x7fa9d41457f0 2024-03-29T11:16:43.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.080+0000 7fa9bbfff640 1 -- 172.21.15.18:0/2738488799 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 822411873 0 0) 0x7fa9d4141ad0 con 0x7fa9d413f650 2024-03-29T11:16:43.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.080+0000 7fa9bbfff640 1 -- 172.21.15.18:0/2738488799 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9d41417f0 con 0x7fa9d413f650 2024-03-29T11:16:43.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.080+0000 7fa9bbfff640 1 -- 172.21.15.18:0/2738488799 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fa9c4004940 con 0x7fa9d413f650 2024-03-29T11:16:43.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.080+0000 7fa9bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:43.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.080+0000 7fa9bbfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:16:43.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.080+0000 7fa9bbfff640 10 monclient(hunting): dump: 2024-03-29T11:16:43.068 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:16:43.068 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:43.068 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:43.068 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:43.068 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:43.068 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:43.068 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:16:43.068 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:16:43.068 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:16:43.069 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:43.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.081+0000 7fa9bbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:43.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.081+0000 7fa9bbfff640 1 -- 172.21.15.18:0/2738488799 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3477597466 0 0) 0x7fa9c0004eb0 con 0x7fa9d40a9d70 2024-03-29T11:16:43.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.081+0000 7fa9bbfff640 1 -- 172.21.15.18:0/2738488799 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9d4141ad0 con 0x7fa9d40a9d70 2024-03-29T11:16:43.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.081+0000 7fa9bbfff640 1 -- 172.21.15.18:0/2738488799 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1871494141 0 0) 0x7fa9d4141510 con 0x7fa9d41457f0 2024-03-29T11:16:43.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.081+0000 7fa9bbfff640 1 -- 172.21.15.18:0/2738488799 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9c0004eb0 con 0x7fa9d41457f0 2024-03-29T11:16:43.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.081+0000 7fa9bbfff640 1 -- 172.21.15.18:0/2738488799 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fa9c8001910 con 0x7fa9d40a9d70 2024-03-29T11:16:43.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.081+0000 7fa9bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:43.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.081+0000 7fa9bbfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:43.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.081+0000 7fa9bbfff640 10 monclient(hunting): dump: 2024-03-29T11:16:43.069 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:16:43.069 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:43.070 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:43.070 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:43.070 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:43.070 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:43.070 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:16:43.070 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:16:43.070 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:16:43.070 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:43.070 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.081+0000 7fa9bbfff640 1 -- 172.21.15.18:0/2738488799 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fa9cc0019e0 con 0x7fa9d41457f0 2024-03-29T11:16:43.070 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.081+0000 7fa9bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:43.070 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.081+0000 7fa9bbfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:16:43.070 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.081+0000 7fa9bbfff640 10 monclient(hunting): dump: 2024-03-29T11:16:43.070 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:16:43.071 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:43.071 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:43.071 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:43.071 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:43.071 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:43.071 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:16:43.071 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:16:43.071 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:16:43.071 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:43.071 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.081+0000 7fa9bbfff640 1 -- 172.21.15.18:0/2738488799 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 491832128 0 0) 0x7fa9d4141ad0 con 0x7fa9d40a9d70 2024-03-29T11:16:43.071 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.081+0000 7fa9bbfff640 1 -- 172.21.15.18:0/2738488799 >> v1:172.21.15.70:6789/0 conn(0x7fa9d41457f0 legacy=0x7fa9d41350e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:43.071 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.081+0000 7fa9bbfff640 1 -- 172.21.15.18:0/2738488799 >> v1:172.21.15.18:6789/0 conn(0x7fa9d413f650 legacy=0x7fa9d41421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:43.071 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.081+0000 7fa9bbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:16:43.071 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.081+0000 7fa9bbfff640 1 monclient: found mon.c 2024-03-29T11:16:43.072 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.081+0000 7fa9bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:43.072 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.081+0000 7fa9bbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:16:43.072 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.081+0000 7fa9bbfff640 1 -- 172.21.15.18:0/2738488799 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d41428e0 con 0x7fa9d40a9d70 2024-03-29T11:16:43.072 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.081+0000 7fa9bbfff640 10 monclient: _finish_auth 0 2024-03-29T11:16:43.072 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.081+0000 7fa9bbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:16:43.072 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.081+0000 7fa9bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:43.072 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.081+0000 7fa9da243640 5 monclient: authenticate success, global_id 4547 2024-03-29T11:16:43.072 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.081+0000 7fa9da243640 10 monclient: _renew_subs 2024-03-29T11:16:43.072 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.081+0000 7fa9da243640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:16:43.072 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.081+0000 7fa9da243640 1 -- 172.21.15.18:0/2738488799 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9d41357f0 con 0x7fa9d40a9d70 2024-03-29T11:16:43.072 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.081+0000 7fa9da243640 10 client.4547.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:43.072 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.081+0000 7fa9da243640 10 monclient: _renew_subs 2024-03-29T11:16:43.072 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.081+0000 7fa9da243640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:16:43.072 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.081+0000 7fa9da243640 1 -- 172.21.15.18:0/2738488799 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9d41401a0 con 0x7fa9d40a9d70 2024-03-29T11:16:43.073 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.081+0000 7fa9da243640 1 librados: init done 2024-03-29T11:16:43.073 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.082+0000 7fa9bbfff640 1 -- 172.21.15.18:0/2738488799 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9c8001a70 con 0x7fa9d40a9d70 2024-03-29T11:16:43.073 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.082+0000 7fa9bbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:43.073 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.082+0000 7fa9bbfff640 1 -- 172.21.15.18:0/2738488799 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fa9c8003810 con 0x7fa9d40a9d70 2024-03-29T11:16:43.073 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.082+0000 7fa9bbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:43.073 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.082+0000 7fa9bbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:43.073 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.082+0000 7fa9bbfff640 10 monclient: dump: 2024-03-29T11:16:43.073 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:16:43.073 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:43.073 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:43.073 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:43.073 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:43.073 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:43.074 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:16:43.074 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:16:43.074 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:16:43.074 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:43.074 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.084+0000 7fa9da243640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:43.074 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.084+0000 7fa9bbfff640 1 -- 172.21.15.18:0/2738488799 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7fa9c80197e0 con 0x7fa9d40a9d70 2024-03-29T11:16:43.074 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.084+0000 7fa9da243640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:43.074 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.084+0000 7fa9da243640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:16:43.074 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.084+0000 7fa9da243640 1 -- 172.21.15.18:0/2738488799 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9d4075f10 con 0x7fa9d40a9d70 2024-03-29T11:16:43.074 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.085+0000 7fa9d1ffb640 4 set_mon_vals no callback set 2024-03-29T11:16:43.075 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.086+0000 7fa9d2ffd640 10 monclient: get_auth_request con 0x7fa9c006e1b0 auth_method 0 2024-03-29T11:16:43.075 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.086+0000 7fa9bbfff640 1 -- 172.21.15.18:0/2738488799 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7fa9c8083880 con 0x7fa9d40a9d70 2024-03-29T11:16:43.075 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.086+0000 7fa9bbfff640 10 client.4547.objecter ms_dispatch 0x7fa9d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:43.075 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.086+0000 7fa9bbfff640 3 client.4547.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:43.075 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.086+0000 7fa9bbfff640 3 client.4547.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:43.075 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.086+0000 7fa9bbfff640 20 client.4547.objecter dump_active .. 0 homeless 2024-03-29T11:16:43.076 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.086+0000 7fa9bbfff640 10 client.4547.objecter ms_handle_connect 0x7fa9c006e1b0 2024-03-29T11:16:43.076 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.094+0000 7fa9bbfff640 1 -- 172.21.15.18:0/2738488799 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa9c804f160 con 0x7fa9d40a9d70 2024-03-29T11:16:43.076 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.094+0000 7fa9bbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:43.076 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.094+0000 7fa9bbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:43.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.482+0000 7fa9da243640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:43.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.482+0000 7fa9da243640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:43.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.482+0000 7fa9da243640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:16:43.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.482+0000 7fa9da243640 1 -- 172.21.15.18:0/2738488799 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9d4142a70 con 0x7fa9d40a9d70 2024-03-29T11:16:43.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.483+0000 7fa9bbfff640 1 -- 172.21.15.18:0/2738488799 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7fa9c8051090 con 0x7fa9d40a9d70 2024-03-29T11:16:43.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.483+0000 7fa9bbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:43.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.483+0000 7fa9bbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:43.462 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:16:43.462 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:43.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.487+0000 7fa9da243640 10 librados: watch_flush enter 2024-03-29T11:16:43.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.487+0000 7fa9da243640 10 librados: watch_flush exit 2024-03-29T11:16:43.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.487+0000 7fa9da243640 20 client.4547.objecter shutdown clearing up homeless session... 2024-03-29T11:16:43.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.487+0000 7fa9da243640 10 client.4547.objecter successfully canceled tick 2024-03-29T11:16:43.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.488+0000 7fa9da243640 1 -- 172.21.15.18:0/2738488799 >> v1:172.21.15.70:6816/3101871562 conn(0x7fa9c006e1b0 legacy=0x7fa9c0070670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:43.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.488+0000 7fa9da243640 10 monclient: shutdown 2024-03-29T11:16:43.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.488+0000 7fa9da243640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:43.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.488+0000 7fa9da243640 1 -- 172.21.15.18:0/2738488799 >> v1:172.21.15.18:6790/0 conn(0x7fa9d40a9d70 legacy=0x7fa9d41db420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:43.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.489+0000 7fa9da243640 1 -- 172.21.15.18:0/2738488799 shutdown_connections 2024-03-29T11:16:43.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.489+0000 7fa9da243640 1 -- 172.21.15.18:0/2738488799 >> 172.21.15.18:0/2738488799 conn(0x7fa9d4066a00 msgr2=0x7fa9d40b3950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:43.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.489+0000 7fa9da243640 1 -- 172.21.15.18:0/2738488799 shutdown_connections 2024-03-29T11:16:43.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.489+0000 7fa9da243640 1 -- 172.21.15.18:0/2738488799 wait complete. 2024-03-29T11:16:43.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:43.489+0000 7fa9da243640 1 librados: shutdown 2024-03-29T11:16:43.483 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:50.290 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:50.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.466+0000 7f4513bed640 10 monclient: get_monmap_and_config 2024-03-29T11:16:50.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.466+0000 7f4513bed640 10 monclient: build_initial_monmap 2024-03-29T11:16:50.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.466+0000 7f4513bed640 10 monclient: monmap: 2024-03-29T11:16:50.443 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:16:50.443 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:50.443 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:16:50.466970+0000 2024-03-29T11:16:50.444 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:16:50.466970+0000 2024-03-29T11:16:50.444 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:50.444 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:50.444 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:16:50.444 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:16:50.444 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:16:50.444 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:50.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.466+0000 7f4513bed640 1 Processor -- start 2024-03-29T11:16:50.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.467+0000 7f4513bed640 1 -- start start 2024-03-29T11:16:50.445 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.467+0000 7f4513bed640 10 monclient: init 2024-03-29T11:16:50.457 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.468+0000 7f4513bed640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:50.457 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.468+0000 7f4513bed640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:16:50.457 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.468+0000 7f4513bed640 10 monclient(hunting): picked mon.noname-a con 0x7f450c1422b0 addr v1:172.21.15.18:6789/0 2024-03-29T11:16:50.457 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.468+0000 7f4513bed640 10 monclient(hunting): picked mon.noname-c con 0x7f450c142690 addr v1:172.21.15.70:6789/0 2024-03-29T11:16:50.457 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.468+0000 7f4513bed640 10 monclient(hunting): picked mon.noname-b con 0x7f450c0b17d0 addr v1:172.21.15.18:6790/0 2024-03-29T11:16:50.457 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.468+0000 7f4513bed640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f450c142c80 con 0x7f450c1422b0 2024-03-29T11:16:50.457 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.468+0000 7f4513bed640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f450c0aae40 con 0x7f450c142690 2024-03-29T11:16:50.457 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.468+0000 7f4513bed640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f450c13fbd0 con 0x7f450c0b17d0 2024-03-29T11:16:50.457 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.468+0000 7f4513bed640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:50.458 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.468+0000 7f4513bed640 10 monclient(hunting): authenticate will time out at 1236.740279s 2024-03-29T11:16:50.458 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.468+0000 7f4511989640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f450c1422b0 0x7f450c13f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:60736/0 (socket says 172.21.15.18:60736) 2024-03-29T11:16:50.458 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.468+0000 7f4511989640 1 -- 172.21.15.18:0/2059507577 learned_addr learned my addr 172.21.15.18:0/2059507577 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:16:50.458 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.469+0000 7f4510987640 1 -- 172.21.15.18:0/2059507577 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3540167722 0 0) 0x7f450c0aae40 con 0x7f450c142690 2024-03-29T11:16:50.458 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.469+0000 7f4510987640 10 monclient(hunting): _negotiate 2024-03-29T11:16:50.458 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.469+0000 7f4510987640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:50.458 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.469+0000 7f4510987640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:50.458 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.469+0000 7f4510987640 10 monclient(hunting): my global_id is 4450 2024-03-29T11:16:50.458 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.469+0000 7f4510987640 1 -- 172.21.15.18:0/2059507577 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44f8003520 con 0x7f450c142690 2024-03-29T11:16:50.458 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.469+0000 7f4510987640 1 -- 172.21.15.18:0/2059507577 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2083384368 0 0) 0x7f450c13fbd0 con 0x7f450c0b17d0 2024-03-29T11:16:50.459 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.469+0000 7f4510987640 10 monclient(hunting): _negotiate 2024-03-29T11:16:50.459 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.469+0000 7f4510987640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:50.459 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.469+0000 7f4510987640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:50.459 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.469+0000 7f4510987640 10 monclient(hunting): my global_id is 4550 2024-03-29T11:16:50.459 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.469+0000 7f4510987640 1 -- 172.21.15.18:0/2059507577 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f450c0aae40 con 0x7f450c0b17d0 2024-03-29T11:16:50.459 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.469+0000 7f4510987640 1 -- 172.21.15.18:0/2059507577 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2631768 0 0) 0x7f450c142c80 con 0x7f450c1422b0 2024-03-29T11:16:50.459 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.469+0000 7f4510987640 10 monclient(hunting): _negotiate 2024-03-29T11:16:50.459 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.469+0000 7f4510987640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:50.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.469+0000 7f4510987640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:50.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.469+0000 7f4510987640 10 monclient(hunting): my global_id is 4482 2024-03-29T11:16:50.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.469+0000 7f4510987640 1 -- 172.21.15.18:0/2059507577 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f450c13fbd0 con 0x7f450c1422b0 2024-03-29T11:16:50.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.469+0000 7f4510987640 1 -- 172.21.15.18:0/2059507577 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2077867293 0 0) 0x7f450c0aae40 con 0x7f450c0b17d0 2024-03-29T11:16:50.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.470+0000 7f4510987640 1 -- 172.21.15.18:0/2059507577 >> v1:172.21.15.70:6789/0 conn(0x7f450c142690 legacy=0x7f450c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:50.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.470+0000 7f4510987640 1 -- 172.21.15.18:0/2059507577 >> v1:172.21.15.18:6789/0 conn(0x7f450c1422b0 legacy=0x7f450c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:50.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.470+0000 7f4510987640 10 monclient: _finish_hunting 0 2024-03-29T11:16:50.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.470+0000 7f4510987640 1 monclient: found mon.noname-b 2024-03-29T11:16:50.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.470+0000 7f4510987640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:50.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.470+0000 7f4510987640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:16:50.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.470+0000 7f4510987640 1 -- 172.21.15.18:0/2059507577 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f450c151940 con 0x7f450c0b17d0 2024-03-29T11:16:50.460 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.470+0000 7f4510987640 10 monclient: _finish_auth 0 2024-03-29T11:16:50.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.470+0000 7f4510987640 10 monclient: _check_auth_tickets 2024-03-29T11:16:50.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.470+0000 7f4510987640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:50.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.470+0000 7f4510987640 1 -- 172.21.15.18:0/2059507577 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f44fc001670 con 0x7f450c0b17d0 2024-03-29T11:16:50.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.470+0000 7f4510987640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:50.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.470+0000 7f4510987640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:16:50.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.470+0000 7f4510987640 10 monclient: dump: 2024-03-29T11:16:50.461 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:16:50.461 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:50.461 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:50.461 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:50.461 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:50.461 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:50.461 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:16:50.461 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:16:50.462 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:16:50.462 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:50.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.470+0000 7f4513bed640 5 monclient: authenticate success, global_id 4550 2024-03-29T11:16:50.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.470+0000 7f4513bed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:50.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.470+0000 7f4510987640 1 -- 172.21.15.18:0/2059507577 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44fc0017d0 con 0x7f450c0b17d0 2024-03-29T11:16:50.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.470+0000 7f4510987640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:50.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.470+0000 7f4513bed640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:50.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.470+0000 7f4513bed640 4 set_mon_vals no callback set 2024-03-29T11:16:50.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.470+0000 7f4510987640 1 -- 172.21.15.18:0/2059507577 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f44fc001b60 con 0x7f450c0b17d0 2024-03-29T11:16:50.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.470+0000 7f4510987640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:50.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.470+0000 7f4510987640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:50.462 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.470+0000 7f4510987640 10 monclient: dump: 2024-03-29T11:16:50.462 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:16:50.462 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:50.463 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:50.463 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:50.463 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:50.463 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:50.463 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:16:50.463 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:16:50.463 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:16:50.463 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:50.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.471+0000 7f4513bed640 10 monclient: shutdown 2024-03-29T11:16:50.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.471+0000 7f4513bed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:50.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.471+0000 7f4513bed640 1 -- 172.21.15.18:0/2059507577 >> v1:172.21.15.18:6790/0 conn(0x7f450c0b17d0 legacy=0x7f450c0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:50.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.471+0000 7f4513bed640 1 -- 172.21.15.18:0/2059507577 shutdown_connections 2024-03-29T11:16:50.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.471+0000 7f4513bed640 1 -- 172.21.15.18:0/2059507577 >> 172.21.15.18:0/2059507577 conn(0x7f450c066a00 msgr2=0x7f450c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:50.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.471+0000 7f4513bed640 1 -- 172.21.15.18:0/2059507577 shutdown_connections 2024-03-29T11:16:50.463 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.471+0000 7f4513bed640 1 -- 172.21.15.18:0/2059507577 wait complete. 2024-03-29T11:16:50.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.472+0000 7f4513bed640 10 monclient: build_initial_monmap 2024-03-29T11:16:50.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.472+0000 7f4513bed640 10 monclient: monmap: 2024-03-29T11:16:50.464 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:16:50.464 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:50.464 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:16:50.464 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:16:50.464 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:50.464 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:50.464 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:16:50.464 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:16:50.464 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:16:50.464 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:50.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.473+0000 7f4513bed640 1 librados: starting msgr at 2024-03-29T11:16:50.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.473+0000 7f4513bed640 1 librados: starting objecter 2024-03-29T11:16:50.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.473+0000 7f4513bed640 1 Processor -- start 2024-03-29T11:16:50.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.473+0000 7f4513bed640 1 -- start start 2024-03-29T11:16:50.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.473+0000 7f4513bed640 1 librados: setting wanted keys 2024-03-29T11:16:50.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.473+0000 7f4513bed640 1 librados: calling monclient init 2024-03-29T11:16:50.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.473+0000 7f4513bed640 10 monclient: init 2024-03-29T11:16:50.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.474+0000 7f4513bed640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:50.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.474+0000 7f4513bed640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:16:50.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.474+0000 7f4513bed640 10 monclient(hunting): picked mon.noname-b con 0x7f450c0b17d0 addr v1:172.21.15.70:6789/0 2024-03-29T11:16:50.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.474+0000 7f4513bed640 10 monclient(hunting): picked mon.noname-a con 0x7f450c1422b0 addr v1:172.21.15.18:6789/0 2024-03-29T11:16:50.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.474+0000 7f4513bed640 10 monclient(hunting): picked mon.noname-c con 0x7f450c142690 addr v1:172.21.15.18:6790/0 2024-03-29T11:16:50.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.474+0000 7f4513bed640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f450c0acf90 con 0x7f450c1422b0 2024-03-29T11:16:50.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.474+0000 7f4513bed640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f450c0ad270 con 0x7f450c0b17d0 2024-03-29T11:16:50.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.474+0000 7f4513bed640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f450c0ab540 con 0x7f450c142690 2024-03-29T11:16:50.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.474+0000 7f4513bed640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:50.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.474+0000 7f4513bed640 10 monclient(hunting): authenticate will time out at 1236.746337s 2024-03-29T11:16:50.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.474+0000 7f4511188640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f450c1422b0 0x7f450c0aba60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:60958/0 (socket says 172.21.15.18:60958) 2024-03-29T11:16:50.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.474+0000 7f4511188640 1 -- 172.21.15.18:0/1932838381 learned_addr learned my addr 172.21.15.18:0/1932838381 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:16:50.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.474+0000 7f45027fc640 10 client.?.objecter ms_handle_connect 0x7f450c1422b0 2024-03-29T11:16:50.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.474+0000 7f45027fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:50.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.474+0000 7f45027fc640 10 client.?.objecter ms_handle_connect 0x7f450c0b17d0 2024-03-29T11:16:50.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.474+0000 7f45027fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:50.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.475+0000 7f45027fc640 10 client.?.objecter ms_handle_connect 0x7f450c142690 2024-03-29T11:16:50.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.475+0000 7f45027fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:50.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.475+0000 7f45027fc640 1 -- 172.21.15.18:0/1932838381 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3097603608 0 0) 0x7f450c0ad270 con 0x7f450c0b17d0 2024-03-29T11:16:50.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.475+0000 7f45027fc640 10 monclient(hunting): _negotiate 2024-03-29T11:16:50.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.475+0000 7f45027fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:50.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.475+0000 7f45027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:50.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.475+0000 7f45027fc640 10 monclient(hunting): my global_id is 4453 2024-03-29T11:16:50.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.475+0000 7f45027fc640 1 -- 172.21.15.18:0/1932838381 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44f0004eb0 con 0x7f450c0b17d0 2024-03-29T11:16:50.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.475+0000 7f45027fc640 1 -- 172.21.15.18:0/1932838381 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 301097662 0 0) 0x7f450c0acf90 con 0x7f450c1422b0 2024-03-29T11:16:50.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.475+0000 7f45027fc640 10 monclient(hunting): _negotiate 2024-03-29T11:16:50.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.475+0000 7f45027fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:50.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.475+0000 7f45027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:50.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.475+0000 7f45027fc640 10 monclient(hunting): my global_id is 4485 2024-03-29T11:16:50.468 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.475+0000 7f45027fc640 1 -- 172.21.15.18:0/1932838381 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f450c0ad270 con 0x7f450c1422b0 2024-03-29T11:16:50.468 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.475+0000 7f45027fc640 1 -- 172.21.15.18:0/1932838381 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1178620736 0 0) 0x7f450c0ab540 con 0x7f450c142690 2024-03-29T11:16:50.468 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.475+0000 7f45027fc640 10 monclient(hunting): _negotiate 2024-03-29T11:16:50.468 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.475+0000 7f45027fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:50.468 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.475+0000 7f45027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:50.468 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.475+0000 7f45027fc640 10 monclient(hunting): my global_id is 4553 2024-03-29T11:16:50.468 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.475+0000 7f45027fc640 1 -- 172.21.15.18:0/1932838381 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f450c0acf90 con 0x7f450c142690 2024-03-29T11:16:50.468 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.475+0000 7f45027fc640 1 -- 172.21.15.18:0/1932838381 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1300696451 0 0) 0x7f44f0004eb0 con 0x7f450c0b17d0 2024-03-29T11:16:50.468 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.476+0000 7f45027fc640 1 -- 172.21.15.18:0/1932838381 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f450c0ab540 con 0x7f450c0b17d0 2024-03-29T11:16:50.468 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.476+0000 7f45027fc640 1 -- 172.21.15.18:0/1932838381 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f45080018f0 con 0x7f450c0b17d0 2024-03-29T11:16:50.468 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.476+0000 7f45027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:50.469 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.476+0000 7f45027fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:16:50.469 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.476+0000 7f45027fc640 10 monclient(hunting): dump: 2024-03-29T11:16:50.469 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:16:50.469 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:50.469 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:50.469 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:50.469 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:50.469 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:50.469 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:16:50.469 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:16:50.469 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:16:50.469 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:50.469 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.476+0000 7f45027fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:50.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.476+0000 7f45027fc640 1 -- 172.21.15.18:0/1932838381 <== mon.1 v1:172.21.15.70:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1523409477 0 0) 0x7f450c0ab540 con 0x7f450c0b17d0 2024-03-29T11:16:50.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.476+0000 7f45027fc640 1 -- 172.21.15.18:0/1932838381 >> v1:172.21.15.18:6790/0 conn(0x7f450c142690 legacy=0x7f450c12ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:50.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.476+0000 7f45027fc640 1 -- 172.21.15.18:0/1932838381 >> v1:172.21.15.18:6789/0 conn(0x7f450c1422b0 legacy=0x7f450c0aba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:50.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.476+0000 7f45027fc640 10 monclient: _finish_hunting 0 2024-03-29T11:16:50.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.476+0000 7f45027fc640 1 monclient: found mon.b 2024-03-29T11:16:50.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.476+0000 7f45027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:50.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.476+0000 7f45027fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:16:50.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.476+0000 7f45027fc640 1 -- 172.21.15.18:0/1932838381 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f450c131cb0 con 0x7f450c0b17d0 2024-03-29T11:16:50.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.476+0000 7f45027fc640 10 monclient: _finish_auth 0 2024-03-29T11:16:50.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.476+0000 7f45027fc640 10 monclient: _check_auth_tickets 2024-03-29T11:16:50.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.476+0000 7f45027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:50.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.476+0000 7f4513bed640 5 monclient: authenticate success, global_id 4453 2024-03-29T11:16:50.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.476+0000 7f4513bed640 10 monclient: _renew_subs 2024-03-29T11:16:50.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.476+0000 7f4513bed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:16:50.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.476+0000 7f4513bed640 1 -- 172.21.15.18:0/1932838381 --> v1:172.21.15.70:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f450c132300 con 0x7f450c0b17d0 2024-03-29T11:16:50.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.477+0000 7f4513bed640 10 client.4453.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:50.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.477+0000 7f4513bed640 10 monclient: _renew_subs 2024-03-29T11:16:50.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.477+0000 7f4513bed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:16:50.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.477+0000 7f4513bed640 1 -- 172.21.15.18:0/1932838381 --> v1:172.21.15.70:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f450c0b75b0 con 0x7f450c0b17d0 2024-03-29T11:16:50.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.477+0000 7f4513bed640 1 librados: init done 2024-03-29T11:16:50.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.477+0000 7f45027fc640 1 -- 172.21.15.18:0/1932838381 <== mon.1 v1:172.21.15.70:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4508001d40 con 0x7f450c0b17d0 2024-03-29T11:16:50.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.477+0000 7f45027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:50.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.477+0000 7f45027fc640 1 -- 172.21.15.18:0/1932838381 <== mon.1 v1:172.21.15.70:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f4508004a80 con 0x7f450c0b17d0 2024-03-29T11:16:50.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.477+0000 7f45027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:50.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.477+0000 7f45027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:16:50.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.477+0000 7f45027fc640 10 monclient: dump: 2024-03-29T11:16:50.472 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:16:50.472 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:50.473 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:50.473 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:50.473 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:50.473 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:50.473 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:16:50.473 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:16:50.473 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:16:50.473 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:50.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.477+0000 7f4503fff640 4 set_mon_vals no callback set 2024-03-29T11:16:50.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.478+0000 7f4513bed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:50.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.478+0000 7f4513bed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:50.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.478+0000 7f4513bed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:16:50.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.478+0000 7f4513bed640 1 -- 172.21.15.18:0/1932838381 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f450c1324f0 con 0x7f450c0b17d0 2024-03-29T11:16:50.474 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.480+0000 7f45027fc640 1 -- 172.21.15.18:0/1932838381 <== mon.1 v1:172.21.15.70:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f4508004be0 con 0x7f450c0b17d0 2024-03-29T11:16:50.475 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.481+0000 7f45027fc640 1 -- 172.21.15.18:0/1932838381 <== mon.1 v1:172.21.15.70:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f4508084a70 con 0x7f450c0b17d0 2024-03-29T11:16:50.475 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.481+0000 7f45027fc640 10 client.4453.objecter ms_dispatch 0x7f450c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:50.475 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.481+0000 7f45027fc640 3 client.4453.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:50.475 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.481+0000 7f45027fc640 3 client.4453.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:50.475 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.481+0000 7f45027fc640 20 client.4453.objecter dump_active .. 0 homeless 2024-03-29T11:16:50.475 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.481+0000 7f4511188640 10 monclient: get_auth_request con 0x7f44f006dea0 auth_method 0 2024-03-29T11:16:50.475 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.482+0000 7f45027fc640 10 client.4453.objecter ms_handle_connect 0x7f44f006dea0 2024-03-29T11:16:50.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.491+0000 7f45027fc640 1 -- 172.21.15.18:0/1932838381 <== mon.1 v1:172.21.15.70:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4508050350 con 0x7f450c0b17d0 2024-03-29T11:16:50.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.491+0000 7f45027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:50.476 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.491+0000 7f45027fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:50.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.883+0000 7f4513bed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:50.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.883+0000 7f4513bed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:50.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.883+0000 7f4513bed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:16:50.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.883+0000 7f4513bed640 1 -- 172.21.15.18:0/1932838381 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f450c0ab6d0 con 0x7f450c0b17d0 2024-03-29T11:16:50.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.885+0000 7f45027fc640 1 -- 172.21.15.18:0/1932838381 <== mon.1 v1:172.21.15.70:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f4508052280 con 0x7f450c0b17d0 2024-03-29T11:16:50.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.885+0000 7f45027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:50.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.885+0000 7f45027fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:50.863 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:16:50.863 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:50.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.889+0000 7f4513bed640 10 librados: watch_flush enter 2024-03-29T11:16:50.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.889+0000 7f4513bed640 10 librados: watch_flush exit 2024-03-29T11:16:50.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.889+0000 7f4513bed640 20 client.4453.objecter shutdown clearing up homeless session... 2024-03-29T11:16:50.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.889+0000 7f4513bed640 10 client.4453.objecter successfully canceled tick 2024-03-29T11:16:50.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.889+0000 7f4513bed640 1 -- 172.21.15.18:0/1932838381 >> v1:172.21.15.70:6816/3101871562 conn(0x7f44f006dea0 legacy=0x7f44f0070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:50.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.889+0000 7f4513bed640 10 monclient: shutdown 2024-03-29T11:16:50.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.889+0000 7f4513bed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:50.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.889+0000 7f4513bed640 1 -- 172.21.15.18:0/1932838381 >> v1:172.21.15.70:6789/0 conn(0x7f450c0b17d0 legacy=0x7f450c1dac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:50.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.889+0000 7f4513bed640 1 -- 172.21.15.18:0/1932838381 shutdown_connections 2024-03-29T11:16:50.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.889+0000 7f4513bed640 1 -- 172.21.15.18:0/1932838381 >> 172.21.15.18:0/1932838381 conn(0x7f450c066a00 msgr2=0x7f450c13dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:50.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.890+0000 7f4513bed640 1 -- 172.21.15.18:0/1932838381 shutdown_connections 2024-03-29T11:16:50.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.890+0000 7f4513bed640 1 -- 172.21.15.18:0/1932838381 wait complete. 2024-03-29T11:16:50.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:50.890+0000 7f4513bed640 1 librados: shutdown 2024-03-29T11:16:50.882 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:16:57.689 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:16:57.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.858+0000 7f26bac74640 10 monclient: get_monmap_and_config 2024-03-29T11:16:57.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.858+0000 7f26bac74640 10 monclient: build_initial_monmap 2024-03-29T11:16:57.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.858+0000 7f26bac74640 10 monclient: monmap: 2024-03-29T11:16:57.837 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:16:57.837 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:57.837 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:16:57.859764+0000 2024-03-29T11:16:57.837 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:16:57.859764+0000 2024-03-29T11:16:57.837 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:57.837 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:57.837 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:16:57.837 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:16:57.837 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:16:57.837 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:57.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.860+0000 7f26bac74640 1 Processor -- start 2024-03-29T11:16:57.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.860+0000 7f26bac74640 1 -- start start 2024-03-29T11:16:57.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.860+0000 7f26bac74640 10 monclient: init 2024-03-29T11:16:57.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.861+0000 7f26bac74640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:57.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.861+0000 7f26bac74640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:16:57.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.861+0000 7f26bac74640 10 monclient(hunting): picked mon.noname-a con 0x7f26b40b0770 addr v1:172.21.15.18:6789/0 2024-03-29T11:16:57.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.861+0000 7f26bac74640 10 monclient(hunting): picked mon.noname-b con 0x7f26b414a700 addr v1:172.21.15.18:6790/0 2024-03-29T11:16:57.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.861+0000 7f26bac74640 10 monclient(hunting): picked mon.noname-c con 0x7f26b40ac5a0 addr v1:172.21.15.70:6789/0 2024-03-29T11:16:57.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.861+0000 7f26bac74640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26b40aba70 con 0x7f26b40b0770 2024-03-29T11:16:57.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.861+0000 7f26bac74640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26b40aecb0 con 0x7f26b40ac5a0 2024-03-29T11:16:57.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.861+0000 7f26bac74640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26b40af150 con 0x7f26b414a700 2024-03-29T11:16:57.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.861+0000 7f26bac74640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:57.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.861+0000 7f26bac74640 10 monclient(hunting): authenticate will time out at 1244.133624s 2024-03-29T11:16:57.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.865+0000 7f26b8a10640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f26b40b0770 0x7f26b4149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:38117/0 (socket says 172.21.15.18:38117) 2024-03-29T11:16:57.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.865+0000 7f26b8a10640 1 -- 172.21.15.18:0/2053229023 learned_addr learned my addr 172.21.15.18:0/2053229023 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:16:57.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.867+0000 7f26b37fe640 1 -- 172.21.15.18:0/2053229023 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1548794759 0 0) 0x7f26b40aba70 con 0x7f26b40b0770 2024-03-29T11:16:57.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.867+0000 7f26b37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:16:57.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.867+0000 7f26b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:57.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.867+0000 7f26b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:57.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.867+0000 7f26b37fe640 10 monclient(hunting): my global_id is 4488 2024-03-29T11:16:57.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.867+0000 7f26b37fe640 1 -- 172.21.15.18:0/2053229023 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26a4003520 con 0x7f26b40b0770 2024-03-29T11:16:57.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.867+0000 7f26b37fe640 1 -- 172.21.15.18:0/2053229023 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1778185049 0 0) 0x7f26b40af150 con 0x7f26b414a700 2024-03-29T11:16:57.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.867+0000 7f26b37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:16:57.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.867+0000 7f26b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:57.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.868+0000 7f26b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:57.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.868+0000 7f26b37fe640 10 monclient(hunting): my global_id is 4556 2024-03-29T11:16:57.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.868+0000 7f26b37fe640 1 -- 172.21.15.18:0/2053229023 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26b40aba70 con 0x7f26b414a700 2024-03-29T11:16:57.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.868+0000 7f26b37fe640 1 -- 172.21.15.18:0/2053229023 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2311251002 0 0) 0x7f26b40aecb0 con 0x7f26b40ac5a0 2024-03-29T11:16:57.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.868+0000 7f26b37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:16:57.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.868+0000 7f26b37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:57.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.868+0000 7f26b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:57.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.868+0000 7f26b37fe640 10 monclient(hunting): my global_id is 4456 2024-03-29T11:16:57.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.868+0000 7f26b37fe640 1 -- 172.21.15.18:0/2053229023 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26b40af150 con 0x7f26b40ac5a0 2024-03-29T11:16:57.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.868+0000 7f26b37fe640 1 -- 172.21.15.18:0/2053229023 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 522617485 0 0) 0x7f26a4003520 con 0x7f26b40b0770 2024-03-29T11:16:57.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.868+0000 7f26b37fe640 1 -- 172.21.15.18:0/2053229023 >> v1:172.21.15.18:6790/0 conn(0x7f26b414a700 legacy=0x7f26b40abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:57.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.868+0000 7f26b37fe640 1 -- 172.21.15.18:0/2053229023 >> v1:172.21.15.70:6789/0 conn(0x7f26b40ac5a0 legacy=0x7f26b4151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:57.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.868+0000 7f26b37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:16:57.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.868+0000 7f26b37fe640 1 monclient: found mon.noname-a 2024-03-29T11:16:57.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.868+0000 7f26b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:57.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.868+0000 7f26b37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-29T11:16:57.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.869+0000 7f26b37fe640 1 -- 172.21.15.18:0/2053229023 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b40ac9d0 con 0x7f26b40b0770 2024-03-29T11:16:57.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.869+0000 7f26b37fe640 10 monclient: _finish_auth 0 2024-03-29T11:16:57.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.869+0000 7f26b37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:16:57.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.869+0000 7f26b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:57.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.869+0000 7f26bac74640 5 monclient: authenticate success, global_id 4488 2024-03-29T11:16:57.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.869+0000 7f26bac74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:57.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.869+0000 7f26b37fe640 1 -- 172.21.15.18:0/2053229023 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f26a8001690 con 0x7f26b40b0770 2024-03-29T11:16:57.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.869+0000 7f26b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:57.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.869+0000 7f26b37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:16:57.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.869+0000 7f26b37fe640 10 monclient: dump: 2024-03-29T11:16:57.852 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:16:57.852 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:57.852 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:57.852 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:57.852 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:57.852 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:57.852 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:16:57.852 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:16:57.852 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:16:57.852 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:57.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.869+0000 7f26bac74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:16:57.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.869+0000 7f26b37fe640 1 -- 172.21.15.18:0/2053229023 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26a80018b0 con 0x7f26b40b0770 2024-03-29T11:16:57.852 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.869+0000 7f26b37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:57.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.869+0000 7f26bac74640 10 monclient: get_monmap_and_config success 2024-03-29T11:16:57.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.869+0000 7f26bac74640 4 set_mon_vals no callback set 2024-03-29T11:16:57.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.870+0000 7f26b37fe640 1 -- 172.21.15.18:0/2053229023 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f26a8001c40 con 0x7f26b40b0770 2024-03-29T11:16:57.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.870+0000 7f26b37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:57.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.870+0000 7f26b37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:16:57.853 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.870+0000 7f26b37fe640 10 monclient: dump: 2024-03-29T11:16:57.853 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:16:57.853 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:57.853 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:57.853 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:57.853 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:57.853 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:57.853 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:16:57.853 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:16:57.854 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:16:57.854 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:57.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.870+0000 7f26bac74640 10 monclient: shutdown 2024-03-29T11:16:57.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.870+0000 7f26bac74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:57.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.870+0000 7f26bac74640 1 -- 172.21.15.18:0/2053229023 >> v1:172.21.15.18:6789/0 conn(0x7f26b40b0770 legacy=0x7f26b4149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:57.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.871+0000 7f26bac74640 1 -- 172.21.15.18:0/2053229023 shutdown_connections 2024-03-29T11:16:57.854 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.871+0000 7f26bac74640 1 -- 172.21.15.18:0/2053229023 >> 172.21.15.18:0/2053229023 conn(0x7f26b406b660 msgr2=0x7f26b406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:57.858 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.871+0000 7f26bac74640 1 -- 172.21.15.18:0/2053229023 shutdown_connections 2024-03-29T11:16:57.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.871+0000 7f26bac74640 1 -- 172.21.15.18:0/2053229023 wait complete. 2024-03-29T11:16:57.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.872+0000 7f26bac74640 10 monclient: build_initial_monmap 2024-03-29T11:16:57.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.872+0000 7f26bac74640 10 monclient: monmap: 2024-03-29T11:16:57.859 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:16:57.859 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:16:57.859 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:16:57.859 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:16:57.859 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:16:57.859 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:57.859 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:16:57.859 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:16:57.859 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:16:57.859 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:57.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.872+0000 7f26bac74640 1 librados: starting msgr at 2024-03-29T11:16:57.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.872+0000 7f26bac74640 1 librados: starting objecter 2024-03-29T11:16:57.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.873+0000 7f26bac74640 1 Processor -- start 2024-03-29T11:16:57.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.873+0000 7f26bac74640 1 -- start start 2024-03-29T11:16:57.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.873+0000 7f26bac74640 1 librados: setting wanted keys 2024-03-29T11:16:57.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.873+0000 7f26bac74640 1 librados: calling monclient init 2024-03-29T11:16:57.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.873+0000 7f26bac74640 10 monclient: init 2024-03-29T11:16:57.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.874+0000 7f26bac74640 10 monclient: _reopen_session rank -1 2024-03-29T11:16:57.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.874+0000 7f26bac74640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:16:57.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.874+0000 7f26bac74640 10 monclient(hunting): picked mon.noname-b con 0x7f26b40ac5a0 addr v1:172.21.15.70:6789/0 2024-03-29T11:16:57.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.874+0000 7f26bac74640 10 monclient(hunting): picked mon.noname-c con 0x7f26b40b0770 addr v1:172.21.15.18:6790/0 2024-03-29T11:16:57.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.874+0000 7f26bac74640 10 monclient(hunting): picked mon.noname-a con 0x7f26b414a700 addr v1:172.21.15.18:6789/0 2024-03-29T11:16:57.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.874+0000 7f26bac74640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26b4126a60 con 0x7f26b414a700 2024-03-29T11:16:57.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.874+0000 7f26bac74640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26b4127640 con 0x7f26b40ac5a0 2024-03-29T11:16:57.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.874+0000 7f26bac74640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26b41284d0 con 0x7f26b40b0770 2024-03-29T11:16:57.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.874+0000 7f26bac74640 10 monclient(hunting): _renew_subs 2024-03-29T11:16:57.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.874+0000 7f26bac74640 10 monclient(hunting): authenticate will time out at 1244.146781s 2024-03-29T11:16:57.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.877+0000 7f26b3fff640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f26b40b0770 0x7f26b41dba50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:48126/0 (socket says 172.21.15.18:48126) 2024-03-29T11:16:57.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.877+0000 7f26b3fff640 1 -- 172.21.15.18:0/423682041 learned_addr learned my addr 172.21.15.18:0/423682041 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:16:57.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.877+0000 7f26b17fa640 10 client.?.objecter ms_handle_connect 0x7f26b40b0770 2024-03-29T11:16:57.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.877+0000 7f26b17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:16:57.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.877+0000 7f26b17fa640 1 -- 172.21.15.18:0/423682041 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1393826354 0 0) 0x7f26b41284d0 con 0x7f26b40b0770 2024-03-29T11:16:57.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.877+0000 7f26b17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:16:57.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.877+0000 7f26b17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:16:57.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.877+0000 7f26b17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:16:57.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.877+0000 7f26b17fa640 10 monclient(hunting): my global_id is 4559 2024-03-29T11:16:57.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.877+0000 7f26b17fa640 1 -- 172.21.15.18:0/423682041 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2698004eb0 con 0x7f26b40b0770 2024-03-29T11:16:57.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.878+0000 7f26b17fa640 1 -- 172.21.15.18:0/423682041 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 364334367 0 0) 0x7f2698004eb0 con 0x7f26b40b0770 2024-03-29T11:16:57.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.878+0000 7f26b17fa640 1 -- 172.21.15.18:0/423682041 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26b41284d0 con 0x7f26b40b0770 2024-03-29T11:16:57.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.878+0000 7f26b17fa640 1 -- 172.21.15.18:0/423682041 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f26a0001910 con 0x7f26b40b0770 2024-03-29T11:16:57.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.878+0000 7f26b17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:57.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.878+0000 7f26b17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:16:57.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.878+0000 7f26b17fa640 10 monclient(hunting): dump: 2024-03-29T11:16:57.862 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:16:57.862 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:57.862 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:57.862 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:57.862 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:57.862 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:57.862 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:16:57.862 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:16:57.863 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:16:57.863 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:57.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.878+0000 7f26b17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:16:57.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.878+0000 7f26b17fa640 1 -- 172.21.15.18:0/423682041 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 298297411 0 0) 0x7f26b41284d0 con 0x7f26b40b0770 2024-03-29T11:16:57.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.878+0000 7f26b17fa640 1 -- 172.21.15.18:0/423682041 >> v1:172.21.15.70:6789/0 conn(0x7f26b40ac5a0 legacy=0x7f26b4126350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:16:57.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.878+0000 7f26b17fa640 1 -- 172.21.15.18:0/423682041 >> v1:172.21.15.18:6789/0 conn(0x7f26b414a700 legacy=0x7f26b41e6200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:16:57.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.880+0000 7f26b17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:16:57.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.880+0000 7f26b17fa640 1 monclient: found mon.c 2024-03-29T11:16:57.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.880+0000 7f26b17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:16:57.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.880+0000 7f26b17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:16:57.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.880+0000 7f26b17fa640 1 -- 172.21.15.18:0/423682041 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26b4128740 con 0x7f26b40b0770 2024-03-29T11:16:57.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.880+0000 7f26b17fa640 10 monclient: _finish_auth 0 2024-03-29T11:16:57.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.880+0000 7f26b17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:16:57.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.880+0000 7f26b17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:16:57.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.880+0000 7f26bac74640 5 monclient: authenticate success, global_id 4559 2024-03-29T11:16:57.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.880+0000 7f26bac74640 10 monclient: _renew_subs 2024-03-29T11:16:57.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.880+0000 7f26bac74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:16:57.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.880+0000 7f26bac74640 1 -- 172.21.15.18:0/423682041 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26b41270d0 con 0x7f26b40b0770 2024-03-29T11:16:57.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.880+0000 7f26bac74640 10 client.4559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:16:57.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.880+0000 7f26bac74640 10 monclient: _renew_subs 2024-03-29T11:16:57.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.880+0000 7f26bac74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:16:57.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.880+0000 7f26bac74640 1 -- 172.21.15.18:0/423682041 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f26b41dc340 con 0x7f26b40b0770 2024-03-29T11:16:57.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.880+0000 7f26bac74640 1 librados: init done 2024-03-29T11:16:57.866 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.882+0000 7f26b17fa640 1 -- 172.21.15.18:0/423682041 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26a00035f0 con 0x7f26b40b0770 2024-03-29T11:16:57.866 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.882+0000 7f26b17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:16:57.866 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.882+0000 7f269effd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:57.866 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.882+0000 7f26b27fc640 4 set_mon_vals no callback set 2024-03-29T11:16:57.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.882+0000 7f26b17fa640 1 -- 172.21.15.18:0/423682041 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f26a0003a40 con 0x7f26b40b0770 2024-03-29T11:16:57.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.882+0000 7f26b17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:16:57.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.882+0000 7f26b17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:16:57.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.882+0000 7f26b17fa640 10 monclient: dump: 2024-03-29T11:16:57.867 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:16:57.867 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:16:57.867 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:57.867 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:16:57.867 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:16:57.867 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:16:57.868 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:16:57.868 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:16:57.868 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:16:57.868 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:16:57.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.883+0000 7f269effd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:57.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.883+0000 7f269effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:16:57.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.883+0000 7f269effd640 1 -- 172.21.15.18:0/423682041 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26b40ab560 con 0x7f26b40b0770 2024-03-29T11:16:57.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.885+0000 7f26b17fa640 1 -- 172.21.15.18:0/423682041 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f26a0003ba0 con 0x7f26b40b0770 2024-03-29T11:16:57.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.886+0000 7f26b17fa640 1 -- 172.21.15.18:0/423682041 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f26a0084a00 con 0x7f26b40b0770 2024-03-29T11:16:57.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.886+0000 7f26b17fa640 10 client.4559.objecter ms_dispatch 0x7f26b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:16:57.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.886+0000 7f26b17fa640 3 client.4559.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:16:57.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.886+0000 7f26b17fa640 3 client.4559.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:16:57.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.886+0000 7f26b17fa640 20 client.4559.objecter dump_active .. 0 homeless 2024-03-29T11:16:57.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.887+0000 7f26b8a10640 10 monclient: get_auth_request con 0x7f269806db30 auth_method 0 2024-03-29T11:16:57.869 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.887+0000 7f26b17fa640 10 client.4559.objecter ms_handle_connect 0x7f269806db30 2024-03-29T11:16:57.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.894+0000 7f26b17fa640 1 -- 172.21.15.18:0/423682041 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f26a0050340 con 0x7f26b40b0770 2024-03-29T11:16:57.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.894+0000 7f26b17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:16:57.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:57.894+0000 7f26b17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:16:58.223 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:58.245+0000 7f269effd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:58.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:58.245+0000 7f269effd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:58.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:58.245+0000 7f269effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:16:58.224 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:58.246+0000 7f269effd640 1 -- 172.21.15.18:0/423682041 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f26b41dc640 con 0x7f26b40b0770 2024-03-29T11:16:58.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:58.247+0000 7f26b17fa640 1 -- 172.21.15.18:0/423682041 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f26b41dc640 con 0x7f26b40b0770 2024-03-29T11:16:58.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:58.247+0000 7f26b17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:16:58.225 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:58.247+0000 7f26b17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:16:58.227 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:16:58.227 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:16:58.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:58.252+0000 7f269effd640 10 librados: watch_flush enter 2024-03-29T11:16:58.230 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:58.252+0000 7f269effd640 10 librados: watch_flush exit 2024-03-29T11:16:58.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:58.252+0000 7f269effd640 20 client.4559.objecter shutdown clearing up homeless session... 2024-03-29T11:16:58.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:58.252+0000 7f269effd640 10 client.4559.objecter successfully canceled tick 2024-03-29T11:16:58.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:58.252+0000 7f269effd640 1 -- 172.21.15.18:0/423682041 >> v1:172.21.15.70:6816/3101871562 conn(0x7f269806db30 legacy=0x7f269806fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:58.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:58.252+0000 7f269effd640 10 monclient: shutdown 2024-03-29T11:16:58.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:58.252+0000 7f269effd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:16:58.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:58.252+0000 7f269effd640 1 -- 172.21.15.18:0/423682041 >> v1:172.21.15.18:6790/0 conn(0x7f26b40b0770 legacy=0x7f26b41dba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:16:58.231 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:58.252+0000 7f269effd640 1 -- 172.21.15.18:0/423682041 shutdown_connections 2024-03-29T11:16:58.232 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:58.252+0000 7f269effd640 1 -- 172.21.15.18:0/423682041 >> 172.21.15.18:0/423682041 conn(0x7f26b406b660 msgr2=0x7f26b40b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:16:58.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:58.256+0000 7f269effd640 1 -- 172.21.15.18:0/423682041 shutdown_connections 2024-03-29T11:16:58.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:58.257+0000 7f269effd640 1 -- 172.21.15.18:0/423682041 wait complete. 2024-03-29T11:16:58.234 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:16:58.257+0000 7f269effd640 1 librados: shutdown 2024-03-29T11:16:58.252 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:05.060 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:05.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.220+0000 7f7b72c8d640 10 monclient: get_monmap_and_config 2024-03-29T11:17:05.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.220+0000 7f7b72c8d640 10 monclient: build_initial_monmap 2024-03-29T11:17:05.197 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.220+0000 7f7b72c8d640 10 monclient: monmap: 2024-03-29T11:17:05.197 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:17:05.197 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:05.197 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:17:05.220960+0000 2024-03-29T11:17:05.197 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:17:05.220960+0000 2024-03-29T11:17:05.197 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:05.198 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:05.198 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:17:05.198 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:17:05.198 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:17:05.198 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:05.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.221+0000 7f7b72c8d640 1 Processor -- start 2024-03-29T11:17:05.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.221+0000 7f7b72c8d640 1 -- start start 2024-03-29T11:17:05.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.221+0000 7f7b72c8d640 10 monclient: init 2024-03-29T11:17:05.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.222+0000 7f7b72c8d640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:05.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.222+0000 7f7b72c8d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:17:05.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.222+0000 7f7b72c8d640 10 monclient(hunting): picked mon.noname-c con 0x7f7b6c071820 addr v1:172.21.15.70:6789/0 2024-03-29T11:17:05.199 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.222+0000 7f7b72c8d640 10 monclient(hunting): picked mon.noname-a con 0x7f7b6c0af660 addr v1:172.21.15.18:6789/0 2024-03-29T11:17:05.200 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.222+0000 7f7b70a29640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f7b6c071820 0x7f7b6c142090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.18:47160/0 (socket says 172.21.15.18:47160) 2024-03-29T11:17:05.200 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.222+0000 7f7b70a29640 1 -- 172.21.15.18:0/2382441577 learned_addr learned my addr 172.21.15.18:0/2382441577 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:17:05.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.223+0000 7f7b72c8d640 10 monclient(hunting): picked mon.noname-b con 0x7f7b6c06ab70 addr v1:172.21.15.18:6790/0 2024-03-29T11:17:05.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.223+0000 7f7b72c8d640 1 -- 172.21.15.18:0/2382441577 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b6c0729c0 con 0x7f7b6c0af660 2024-03-29T11:17:05.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.223+0000 7f7b72c8d640 1 -- 172.21.15.18:0/2382441577 --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b6c0afaf0 con 0x7f7b6c071820 2024-03-29T11:17:05.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.223+0000 7f7b72c8d640 1 -- 172.21.15.18:0/2382441577 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b6c0b1110 con 0x7f7b6c06ab70 2024-03-29T11:17:05.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.223+0000 7f7b72c8d640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:05.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.223+0000 7f7b72c8d640 10 monclient(hunting): authenticate will time out at 1251.495062s 2024-03-29T11:17:05.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.223+0000 7f7b6b7fe640 1 -- 172.21.15.18:0/2382441577 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2177018420 0 0) 0x7f7b6c0afaf0 con 0x7f7b6c071820 2024-03-29T11:17:05.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.223+0000 7f7b6b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:17:05.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.223+0000 7f7b6b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:05.201 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.223+0000 7f7b6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:05.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.223+0000 7f7b6b7fe640 10 monclient(hunting): my global_id is 4459 2024-03-29T11:17:05.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.223+0000 7f7b6b7fe640 1 -- 172.21.15.18:0/2382441577 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b5c003520 con 0x7f7b6c071820 2024-03-29T11:17:05.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.223+0000 7f7b6b7fe640 1 -- 172.21.15.18:0/2382441577 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3087293118 0 0) 0x7f7b6c0b1110 con 0x7f7b6c06ab70 2024-03-29T11:17:05.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.223+0000 7f7b6b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:17:05.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.223+0000 7f7b6b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:05.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.223+0000 7f7b6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:05.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.223+0000 7f7b6b7fe640 10 monclient(hunting): my global_id is 4562 2024-03-29T11:17:05.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.223+0000 7f7b6b7fe640 1 -- 172.21.15.18:0/2382441577 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b5c003840 con 0x7f7b6c06ab70 2024-03-29T11:17:05.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.224+0000 7f7b6b7fe640 1 -- 172.21.15.18:0/2382441577 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1306604960 0 0) 0x7f7b5c003520 con 0x7f7b6c071820 2024-03-29T11:17:05.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.224+0000 7f7b6b7fe640 1 -- 172.21.15.18:0/2382441577 >> v1:172.21.15.18:6790/0 conn(0x7f7b6c06ab70 legacy=0x7f7b6c14d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:05.202 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.224+0000 7f7b6b7fe640 1 -- 172.21.15.18:0/2382441577 >> v1:172.21.15.18:6789/0 conn(0x7f7b6c0af660 legacy=0x7f7b6c072180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:17:05.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.225+0000 7f7b6b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:17:05.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.225+0000 7f7b6b7fe640 1 monclient: found mon.noname-c 2024-03-29T11:17:05.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.225+0000 7f7b6b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:05.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.225+0000 7f7b6b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.70:6789/0 2024-03-29T11:17:05.203 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.225+0000 7f7b6b7fe640 1 -- 172.21.15.18:0/2382441577 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b6c0b12a0 con 0x7f7b6c071820 2024-03-29T11:17:05.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.225+0000 7f7b6b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:17:05.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.225+0000 7f7b6b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:17:05.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.225+0000 7f7b6b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:05.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.225+0000 7f7b6b7fe640 1 -- 172.21.15.18:0/2382441577 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f7b60001670 con 0x7f7b6c071820 2024-03-29T11:17:05.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.225+0000 7f7b72c8d640 5 monclient: authenticate success, global_id 4459 2024-03-29T11:17:05.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.225+0000 7f7b72c8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:05.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.225+0000 7f7b6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:05.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.225+0000 7f7b6b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:05.204 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.225+0000 7f7b6b7fe640 10 monclient: dump: 2024-03-29T11:17:05.204 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:17:05.204 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:05.204 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:05.204 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:05.204 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:05.204 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:05.205 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:17:05.205 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:17:05.205 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:17:05.205 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:05.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.225+0000 7f7b72c8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:05.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.225+0000 7f7b6b7fe640 1 -- 172.21.15.18:0/2382441577 <== mon.1 v1:172.21.15.70:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b60001850 con 0x7f7b6c071820 2024-03-29T11:17:05.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.225+0000 7f7b6b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:05.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.225+0000 7f7b6b7fe640 1 -- 172.21.15.18:0/2382441577 <== mon.1 v1:172.21.15.70:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f7b60001be0 con 0x7f7b6c071820 2024-03-29T11:17:05.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.225+0000 7f7b6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:05.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.225+0000 7f7b6b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:17:05.205 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.225+0000 7f7b6b7fe640 10 monclient: dump: 2024-03-29T11:17:05.205 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:17:05.205 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:05.205 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:05.205 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:05.206 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:05.206 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:05.206 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:17:05.206 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:17:05.206 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:17:05.206 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:05.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.226+0000 7f7b72c8d640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:05.206 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.226+0000 7f7b72c8d640 4 set_mon_vals no callback set 2024-03-29T11:17:05.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.227+0000 7f7b72c8d640 10 monclient: shutdown 2024-03-29T11:17:05.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.227+0000 7f7b72c8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:05.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.227+0000 7f7b72c8d640 1 -- 172.21.15.18:0/2382441577 >> v1:172.21.15.70:6789/0 conn(0x7f7b6c071820 legacy=0x7f7b6c142090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:05.213 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.227+0000 7f7b72c8d640 1 -- 172.21.15.18:0/2382441577 shutdown_connections 2024-03-29T11:17:05.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.227+0000 7f7b72c8d640 1 -- 172.21.15.18:0/2382441577 >> 172.21.15.18:0/2382441577 conn(0x7f7b6c0b4930 msgr2=0x7f7b6c0b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:05.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.227+0000 7f7b72c8d640 1 -- 172.21.15.18:0/2382441577 shutdown_connections 2024-03-29T11:17:05.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.227+0000 7f7b72c8d640 1 -- 172.21.15.18:0/2382441577 wait complete. 2024-03-29T11:17:05.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.228+0000 7f7b72c8d640 10 monclient: build_initial_monmap 2024-03-29T11:17:05.214 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.228+0000 7f7b72c8d640 10 monclient: monmap: 2024-03-29T11:17:05.214 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:17:05.214 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:05.214 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:17:05.214 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:17:05.214 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:05.214 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:05.214 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:17:05.214 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:17:05.214 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:17:05.214 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:05.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.229+0000 7f7b72c8d640 1 librados: starting msgr at 2024-03-29T11:17:05.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.229+0000 7f7b72c8d640 1 librados: starting objecter 2024-03-29T11:17:05.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.229+0000 7f7b72c8d640 1 Processor -- start 2024-03-29T11:17:05.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.229+0000 7f7b72c8d640 1 -- start start 2024-03-29T11:17:05.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.229+0000 7f7b72c8d640 1 librados: setting wanted keys 2024-03-29T11:17:05.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.229+0000 7f7b72c8d640 1 librados: calling monclient init 2024-03-29T11:17:05.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.229+0000 7f7b72c8d640 10 monclient: init 2024-03-29T11:17:05.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.230+0000 7f7b72c8d640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:05.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.230+0000 7f7b72c8d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:17:05.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.230+0000 7f7b72c8d640 10 monclient(hunting): picked mon.noname-b con 0x7f7b6c06ab70 addr v1:172.21.15.70:6789/0 2024-03-29T11:17:05.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.230+0000 7f7b72c8d640 10 monclient(hunting): picked mon.noname-a con 0x7f7b6c071820 addr v1:172.21.15.18:6789/0 2024-03-29T11:17:05.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.231+0000 7f7b72c8d640 10 monclient(hunting): picked mon.noname-c con 0x7f7b6c0af660 addr v1:172.21.15.18:6790/0 2024-03-29T11:17:05.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.231+0000 7f7b72c8d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b6c13b110 con 0x7f7b6c071820 2024-03-29T11:17:05.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.231+0000 7f7b72c8d640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b6c13eaa0 con 0x7f7b6c06ab70 2024-03-29T11:17:05.215 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.231+0000 7f7b72c8d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b6c0aabf0 con 0x7f7b6c0af660 2024-03-29T11:17:05.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.231+0000 7f7b72c8d640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:05.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.231+0000 7f7b72c8d640 10 monclient(hunting): authenticate will time out at 1251.503425s 2024-03-29T11:17:05.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.231+0000 7f7b7122a640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f7b6c0af660 0x7f7b6c0a9bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:59324/0 (socket says 172.21.15.18:59324) 2024-03-29T11:17:05.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.231+0000 7f7b7122a640 1 -- 172.21.15.18:0/1812207432 learned_addr learned my addr 172.21.15.18:0/1812207432 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:17:05.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.231+0000 7f7b697fa640 10 client.?.objecter ms_handle_connect 0x7f7b6c0af660 2024-03-29T11:17:05.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.231+0000 7f7b697fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:05.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.232+0000 7f7b697fa640 1 -- 172.21.15.18:0/1812207432 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1450478653 0 0) 0x7f7b6c0aabf0 con 0x7f7b6c0af660 2024-03-29T11:17:05.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.232+0000 7f7b697fa640 10 monclient(hunting): _negotiate 2024-03-29T11:17:05.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.232+0000 7f7b697fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:05.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.232+0000 7f7b697fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:05.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.232+0000 7f7b697fa640 10 monclient(hunting): my global_id is 4565 2024-03-29T11:17:05.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.232+0000 7f7b697fa640 1 -- 172.21.15.18:0/1812207432 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b50004eb0 con 0x7f7b6c0af660 2024-03-29T11:17:05.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.232+0000 7f7b697fa640 1 -- 172.21.15.18:0/1812207432 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2079950467 0 0) 0x7f7b50004eb0 con 0x7f7b6c0af660 2024-03-29T11:17:05.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.232+0000 7f7b697fa640 1 -- 172.21.15.18:0/1812207432 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b6c0aabf0 con 0x7f7b6c0af660 2024-03-29T11:17:05.216 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.232+0000 7f7b697fa640 1 -- 172.21.15.18:0/1812207432 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f7b64001d10 con 0x7f7b6c0af660 2024-03-29T11:17:05.217 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.232+0000 7f7b697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:05.217 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.232+0000 7f7b697fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:05.217 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.232+0000 7f7b697fa640 10 monclient(hunting): dump: 2024-03-29T11:17:05.217 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:17:05.217 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:05.217 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:05.217 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:05.217 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:05.217 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:05.217 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:17:05.217 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:17:05.217 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:17:05.217 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:05.217 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.232+0000 7f7b697fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:05.217 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.232+0000 7f7b697fa640 10 client.?.objecter ms_handle_connect 0x7f7b6c06ab70 2024-03-29T11:17:05.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.232+0000 7f7b697fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:05.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.233+0000 7f7b697fa640 1 -- 172.21.15.18:0/1812207432 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2960890603 0 0) 0x7f7b6c0aabf0 con 0x7f7b6c0af660 2024-03-29T11:17:05.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.233+0000 7f7b697fa640 1 -- 172.21.15.18:0/1812207432 >> v1:172.21.15.70:6789/0 conn(0x7f7b6c06ab70 legacy=0x7f7b6c0acae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:05.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.233+0000 7f7b697fa640 1 -- 172.21.15.18:0/1812207432 >> v1:172.21.15.18:6789/0 conn(0x7f7b6c071820 legacy=0x7f7b6c0a94e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:17:05.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.233+0000 7f7b697fa640 10 monclient: _finish_hunting 0 2024-03-29T11:17:05.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.233+0000 7f7b697fa640 1 monclient: found mon.c 2024-03-29T11:17:05.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.233+0000 7f7b697fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:05.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.233+0000 7f7b697fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:17:05.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.233+0000 7f7b697fa640 1 -- 172.21.15.18:0/1812207432 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b6c13b360 con 0x7f7b6c0af660 2024-03-29T11:17:05.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.234+0000 7f7b697fa640 10 monclient: _finish_auth 0 2024-03-29T11:17:05.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.234+0000 7f7b697fa640 10 monclient: _check_auth_tickets 2024-03-29T11:17:05.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.234+0000 7f7b697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:05.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.234+0000 7f7b72c8d640 5 monclient: authenticate success, global_id 4565 2024-03-29T11:17:05.218 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.234+0000 7f7b72c8d640 10 monclient: _renew_subs 2024-03-29T11:17:05.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.234+0000 7f7b72c8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:17:05.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.234+0000 7f7b72c8d640 1 -- 172.21.15.18:0/1812207432 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b6c13ba60 con 0x7f7b6c0af660 2024-03-29T11:17:05.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.234+0000 7f7b697fa640 1 -- 172.21.15.18:0/1812207432 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b64002010 con 0x7f7b6c0af660 2024-03-29T11:17:05.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.234+0000 7f7b697fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:05.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.234+0000 7f7b697fa640 1 -- 172.21.15.18:0/1812207432 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f7b64003de0 con 0x7f7b6c0af660 2024-03-29T11:17:05.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.234+0000 7f7b697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:05.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.234+0000 7f7b697fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:05.219 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.234+0000 7f7b697fa640 10 monclient: dump: 2024-03-29T11:17:05.219 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:17:05.219 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:05.219 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:05.219 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:05.219 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:05.219 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:05.219 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:17:05.220 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:17:05.220 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:17:05.220 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:05.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.234+0000 7f7b72c8d640 10 client.4565.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:05.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.234+0000 7f7b72c8d640 10 monclient: _renew_subs 2024-03-29T11:17:05.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.234+0000 7f7b72c8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:17:05.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.234+0000 7f7b72c8d640 1 -- 172.21.15.18:0/1812207432 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b6c0ad3d0 con 0x7f7b6c0af660 2024-03-29T11:17:05.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.234+0000 7f7b72c8d640 1 librados: init done 2024-03-29T11:17:05.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.234+0000 7f7b6affd640 4 set_mon_vals no callback set 2024-03-29T11:17:05.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.236+0000 7f7b72c8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:05.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.236+0000 7f7b72c8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:05.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.236+0000 7f7b72c8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:17:05.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.236+0000 7f7b72c8d640 1 -- 172.21.15.18:0/1812207432 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b6c0b3c40 con 0x7f7b6c0af660 2024-03-29T11:17:05.220 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.236+0000 7f7b697fa640 1 -- 172.21.15.18:0/1812207432 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f7b64003f40 con 0x7f7b6c0af660 2024-03-29T11:17:05.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.237+0000 7f7b697fa640 1 -- 172.21.15.18:0/1812207432 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f7b64084e60 con 0x7f7b6c0af660 2024-03-29T11:17:05.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.237+0000 7f7b697fa640 10 client.4565.objecter ms_dispatch 0x7f7b6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:05.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.237+0000 7f7b697fa640 3 client.4565.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:05.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.237+0000 7f7b697fa640 3 client.4565.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:05.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.237+0000 7f7b697fa640 20 client.4565.objecter dump_active .. 0 homeless 2024-03-29T11:17:05.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.237+0000 7f7b70a29640 10 monclient: get_auth_request con 0x7f7b5006dc40 auth_method 0 2024-03-29T11:17:05.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.237+0000 7f7b697fa640 10 client.4565.objecter ms_handle_connect 0x7f7b5006dc40 2024-03-29T11:17:05.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.243+0000 7f7b697fa640 1 -- 172.21.15.18:0/1812207432 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7b640507a0 con 0x7f7b6c0af660 2024-03-29T11:17:05.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.243+0000 7f7b697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:05.222 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.243+0000 7f7b697fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:05.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.547+0000 7f7b72c8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:05.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.547+0000 7f7b72c8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:05.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.547+0000 7f7b72c8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:17:05.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.547+0000 7f7b72c8d640 1 -- 172.21.15.18:0/1812207432 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b6c075f10 con 0x7f7b6c0af660 2024-03-29T11:17:05.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.548+0000 7f7b697fa640 1 -- 172.21.15.18:0/1812207432 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f7b64084510 con 0x7f7b6c0af660 2024-03-29T11:17:05.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.548+0000 7f7b697fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:05.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.548+0000 7f7b697fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:05.527 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:17:05.527 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:05.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.551+0000 7f7b72c8d640 10 librados: watch_flush enter 2024-03-29T11:17:05.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.551+0000 7f7b72c8d640 10 librados: watch_flush exit 2024-03-29T11:17:05.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.551+0000 7f7b72c8d640 20 client.4565.objecter shutdown clearing up homeless session... 2024-03-29T11:17:05.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.551+0000 7f7b72c8d640 10 client.4565.objecter successfully canceled tick 2024-03-29T11:17:05.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.551+0000 7f7b72c8d640 1 -- 172.21.15.18:0/1812207432 >> v1:172.21.15.70:6816/3101871562 conn(0x7f7b5006dc40 legacy=0x7f7b50070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:05.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.551+0000 7f7b72c8d640 10 monclient: shutdown 2024-03-29T11:17:05.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.551+0000 7f7b72c8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:05.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.551+0000 7f7b72c8d640 1 -- 172.21.15.18:0/1812207432 >> v1:172.21.15.18:6790/0 conn(0x7f7b6c0af660 legacy=0x7f7b6c0a9bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:05.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.551+0000 7f7b72c8d640 1 -- 172.21.15.18:0/1812207432 shutdown_connections 2024-03-29T11:17:05.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.552+0000 7f7b72c8d640 1 -- 172.21.15.18:0/1812207432 >> 172.21.15.18:0/1812207432 conn(0x7f7b6c0b4930 msgr2=0x7f7b6c0b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:05.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.552+0000 7f7b72c8d640 1 -- 172.21.15.18:0/1812207432 shutdown_connections 2024-03-29T11:17:05.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.552+0000 7f7b72c8d640 1 -- 172.21.15.18:0/1812207432 wait complete. 2024-03-29T11:17:05.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:05.552+0000 7f7b72c8d640 1 librados: shutdown 2024-03-29T11:17:05.544 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:12.352 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:12.495 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.518+0000 7f72e35bc640 10 monclient: get_monmap_and_config 2024-03-29T11:17:12.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.518+0000 7f72e35bc640 10 monclient: build_initial_monmap 2024-03-29T11:17:12.496 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.518+0000 7f72e35bc640 10 monclient: monmap: 2024-03-29T11:17:12.496 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:17:12.496 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:12.496 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:17:12.519465+0000 2024-03-29T11:17:12.496 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:17:12.519465+0000 2024-03-29T11:17:12.496 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:12.496 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:12.496 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:17:12.496 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:17:12.497 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:17:12.497 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:12.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.519+0000 7f72e35bc640 1 Processor -- start 2024-03-29T11:17:12.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.519+0000 7f72e35bc640 1 -- start start 2024-03-29T11:17:12.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.519+0000 7f72e35bc640 10 monclient: init 2024-03-29T11:17:12.497 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.520+0000 7f72e35bc640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:12.498 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.520+0000 7f72e35bc640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:17:12.500 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.520+0000 7f72e35bc640 10 monclient(hunting): picked mon.noname-c con 0x7f72dc0b0770 addr v1:172.21.15.70:6789/0 2024-03-29T11:17:12.500 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.520+0000 7f72e35bc640 10 monclient(hunting): picked mon.noname-a con 0x7f72dc0ad850 addr v1:172.21.15.18:6789/0 2024-03-29T11:17:12.500 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.520+0000 7f72e35bc640 10 monclient(hunting): picked mon.noname-b con 0x7f72dc13c1b0 addr v1:172.21.15.18:6790/0 2024-03-29T11:17:12.500 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.520+0000 7f72e35bc640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72dc0ac8e0 con 0x7f72dc0ad850 2024-03-29T11:17:12.500 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.520+0000 7f72e35bc640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72dc0ad330 con 0x7f72dc0b0770 2024-03-29T11:17:12.500 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.520+0000 7f72e35bc640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72dc0acfe0 con 0x7f72dc13c1b0 2024-03-29T11:17:12.501 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.520+0000 7f72e35bc640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:12.501 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.520+0000 7f72e35bc640 10 monclient(hunting): authenticate will time out at 1258.792456s 2024-03-29T11:17:12.501 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.523+0000 7f72e1b59640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f72dc13c1b0 0x7f72dc14a760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:43912/0 (socket says 172.21.15.18:43912) 2024-03-29T11:17:12.501 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.523+0000 7f72e1b59640 1 -- 172.21.15.18:0/989305725 learned_addr learned my addr 172.21.15.18:0/989305725 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:17:12.501 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.524+0000 7f72d3fff640 1 -- 172.21.15.18:0/989305725 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2656151687 0 0) 0x7f72dc0acfe0 con 0x7f72dc13c1b0 2024-03-29T11:17:12.501 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.524+0000 7f72d3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:17:12.501 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.524+0000 7f72d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:12.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.524+0000 7f72d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:12.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.524+0000 7f72d3fff640 10 monclient(hunting): my global_id is 4568 2024-03-29T11:17:12.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.524+0000 7f72d3fff640 1 -- 172.21.15.18:0/989305725 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72c8003520 con 0x7f72dc13c1b0 2024-03-29T11:17:12.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.524+0000 7f72d3fff640 1 -- 172.21.15.18:0/989305725 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1284438336 0 0) 0x7f72c8003520 con 0x7f72dc13c1b0 2024-03-29T11:17:12.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.524+0000 7f72d3fff640 1 -- 172.21.15.18:0/989305725 >> v1:172.21.15.70:6789/0 conn(0x7f72dc0b0770 legacy=0x7f72dc072070 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:17:12.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.524+0000 7f72d3fff640 1 -- 172.21.15.18:0/989305725 >> v1:172.21.15.18:6789/0 conn(0x7f72dc0ad850 legacy=0x7f72dc0aaae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:17:12.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.527+0000 7f72d3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:17:12.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.527+0000 7f72d3fff640 1 monclient: found mon.noname-b 2024-03-29T11:17:12.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.527+0000 7f72d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:12.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.527+0000 7f72d3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:17:12.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.527+0000 7f72d3fff640 1 -- 172.21.15.18:0/989305725 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72dc0ab1f0 con 0x7f72dc13c1b0 2024-03-29T11:17:12.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.527+0000 7f72d3fff640 10 monclient: _finish_auth 0 2024-03-29T11:17:12.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.527+0000 7f72d3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:17:12.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.527+0000 7f72d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:12.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.527+0000 7f72d3fff640 1 -- 172.21.15.18:0/989305725 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f72d4001a10 con 0x7f72dc13c1b0 2024-03-29T11:17:12.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.527+0000 7f72d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:12.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.527+0000 7f72d3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:17:12.505 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.527+0000 7f72d3fff640 10 monclient: dump: 2024-03-29T11:17:12.505 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:17:12.505 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:12.505 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:12.505 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:12.505 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:12.505 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:12.505 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:17:12.505 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:17:12.505 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:17:12.505 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:12.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.527+0000 7f72d3fff640 1 -- 172.21.15.18:0/989305725 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f72d4001be0 con 0x7f72dc13c1b0 2024-03-29T11:17:12.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.527+0000 7f72d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:12.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.527+0000 7f72d3fff640 1 -- 172.21.15.18:0/989305725 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f72d4002030 con 0x7f72dc13c1b0 2024-03-29T11:17:12.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.527+0000 7f72d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:12.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.527+0000 7f72d3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:12.506 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.527+0000 7f72d3fff640 10 monclient: dump: 2024-03-29T11:17:12.506 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:17:12.506 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:12.506 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:12.506 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:12.506 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:12.506 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:12.506 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:17:12.506 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:17:12.507 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:17:12.507 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:12.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.527+0000 7f72e35bc640 5 monclient: authenticate success, global_id 4568 2024-03-29T11:17:12.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.527+0000 7f72e35bc640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:12.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.527+0000 7f72e35bc640 4 set_mon_vals no callback set 2024-03-29T11:17:12.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.528+0000 7f72e35bc640 10 monclient: shutdown 2024-03-29T11:17:12.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.528+0000 7f72e35bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:12.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.528+0000 7f72e35bc640 1 -- 172.21.15.18:0/989305725 >> v1:172.21.15.18:6790/0 conn(0x7f72dc13c1b0 legacy=0x7f72dc14a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:12.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.529+0000 7f72e35bc640 1 -- 172.21.15.18:0/989305725 shutdown_connections 2024-03-29T11:17:12.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.529+0000 7f72e35bc640 1 -- 172.21.15.18:0/989305725 >> 172.21.15.18:0/989305725 conn(0x7f72dc06b660 msgr2=0x7f72dc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:12.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.529+0000 7f72e35bc640 1 -- 172.21.15.18:0/989305725 shutdown_connections 2024-03-29T11:17:12.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.529+0000 7f72e35bc640 1 -- 172.21.15.18:0/989305725 wait complete. 2024-03-29T11:17:12.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.530+0000 7f72e35bc640 10 monclient: build_initial_monmap 2024-03-29T11:17:12.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.530+0000 7f72e35bc640 10 monclient: monmap: 2024-03-29T11:17:12.509 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:17:12.510 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:12.510 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:17:12.510 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:17:12.510 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:12.510 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:12.510 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:17:12.510 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:17:12.510 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:17:12.510 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:12.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.531+0000 7f72e35bc640 1 librados: starting msgr at 2024-03-29T11:17:12.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.531+0000 7f72e35bc640 1 librados: starting objecter 2024-03-29T11:17:12.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.531+0000 7f72e35bc640 1 Processor -- start 2024-03-29T11:17:12.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.531+0000 7f72e35bc640 1 -- start start 2024-03-29T11:17:12.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.532+0000 7f72e35bc640 1 librados: setting wanted keys 2024-03-29T11:17:12.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.532+0000 7f72e35bc640 1 librados: calling monclient init 2024-03-29T11:17:12.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.532+0000 7f72e35bc640 10 monclient: init 2024-03-29T11:17:12.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.534+0000 7f72e35bc640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:12.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.534+0000 7f72e35bc640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:17:12.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.534+0000 7f72e35bc640 10 monclient(hunting): picked mon.noname-b con 0x7f72dc0ad850 addr v1:172.21.15.70:6789/0 2024-03-29T11:17:12.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.534+0000 7f72e35bc640 10 monclient(hunting): picked mon.noname-c con 0x7f72dc0b0770 addr v1:172.21.15.18:6790/0 2024-03-29T11:17:12.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.534+0000 7f72e0b57640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f72dc0b0770 0x7f72dc1d33b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:44592/0 (socket says 172.21.15.18:44592) 2024-03-29T11:17:12.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.534+0000 7f72e0b57640 1 -- 172.21.15.18:0/3386163589 learned_addr learned my addr 172.21.15.18:0/3386163589 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:17:12.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.534+0000 7f72d1ffb640 10 client.?.objecter ms_handle_connect 0x7f72dc0b0770 2024-03-29T11:17:12.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.536+0000 7f72d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:12.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.536+0000 7f72e35bc640 10 monclient(hunting): picked mon.noname-a con 0x7f72dc13c1b0 addr v1:172.21.15.18:6789/0 2024-03-29T11:17:12.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.536+0000 7f72e35bc640 1 -- 172.21.15.18:0/3386163589 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72dc1d2e90 con 0x7f72dc13c1b0 2024-03-29T11:17:12.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.536+0000 7f72e35bc640 1 -- 172.21.15.18:0/3386163589 --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72dc1d2ab0 con 0x7f72dc0ad850 2024-03-29T11:17:12.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.536+0000 7f72e35bc640 1 -- 172.21.15.18:0/3386163589 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72dc1d2c40 con 0x7f72dc0b0770 2024-03-29T11:17:12.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.536+0000 7f72e35bc640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:12.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.536+0000 7f72e35bc640 10 monclient(hunting): authenticate will time out at 1258.808881s 2024-03-29T11:17:12.517 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.537+0000 7f72d1ffb640 1 -- 172.21.15.18:0/3386163589 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3646967277 0 0) 0x7f72dc1d2c40 con 0x7f72dc0b0770 2024-03-29T11:17:12.517 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.537+0000 7f72d1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:17:12.517 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.537+0000 7f72d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:12.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.537+0000 7f72d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:12.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.537+0000 7f72d1ffb640 10 monclient(hunting): my global_id is 4571 2024-03-29T11:17:12.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.537+0000 7f72d1ffb640 1 -- 172.21.15.18:0/3386163589 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72c0004eb0 con 0x7f72dc0b0770 2024-03-29T11:17:12.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.539+0000 7f72d1ffb640 1 -- 172.21.15.18:0/3386163589 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2448182079 0 0) 0x7f72c0004eb0 con 0x7f72dc0b0770 2024-03-29T11:17:12.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.539+0000 7f72d1ffb640 1 -- 172.21.15.18:0/3386163589 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72dc1d2c40 con 0x7f72dc0b0770 2024-03-29T11:17:12.518 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.539+0000 7f72d1ffb640 1 -- 172.21.15.18:0/3386163589 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f72cc0018c0 con 0x7f72dc0b0770 2024-03-29T11:17:12.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.539+0000 7f72d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:12.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.539+0000 7f72d1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:12.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.539+0000 7f72d1ffb640 10 monclient(hunting): dump: 2024-03-29T11:17:12.519 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:17:12.519 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:12.519 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:12.519 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:12.519 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:12.519 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:12.519 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:17:12.519 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:17:12.519 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:17:12.519 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:12.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.539+0000 7f72d1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:12.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.539+0000 7f72d1ffb640 1 -- 172.21.15.18:0/3386163589 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2450958086 0 0) 0x7f72dc1d2c40 con 0x7f72dc0b0770 2024-03-29T11:17:12.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.539+0000 7f72d1ffb640 1 -- 172.21.15.18:0/3386163589 >> v1:172.21.15.70:6789/0 conn(0x7f72dc0ad850 legacy=0x7f72dc0b1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:12.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.539+0000 7f72d1ffb640 1 -- 172.21.15.18:0/3386163589 >> v1:172.21.15.18:6789/0 conn(0x7f72dc13c1b0 legacy=0x7f72dc13dfe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:17:12.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.541+0000 7f72d1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:17:12.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.541+0000 7f72d1ffb640 1 monclient: found mon.c 2024-03-29T11:17:12.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.542+0000 7f72d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:12.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.542+0000 7f72d1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:17:12.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.542+0000 7f72d1ffb640 1 -- 172.21.15.18:0/3386163589 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72dc13e910 con 0x7f72dc0b0770 2024-03-29T11:17:12.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.542+0000 7f72d1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:17:12.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.542+0000 7f72d1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:17:12.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.542+0000 7f72d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:12.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.542+0000 7f72d1ffb640 1 -- 172.21.15.18:0/3386163589 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f72cc001a20 con 0x7f72dc0b0770 2024-03-29T11:17:12.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.542+0000 7f72e35bc640 5 monclient: authenticate success, global_id 4571 2024-03-29T11:17:12.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.542+0000 7f72e35bc640 10 monclient: _renew_subs 2024-03-29T11:17:12.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.542+0000 7f72e35bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:17:12.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.542+0000 7f72e35bc640 1 -- 172.21.15.18:0/3386163589 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72dc13eb60 con 0x7f72dc0b0770 2024-03-29T11:17:12.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.542+0000 7f72e35bc640 10 client.4571.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:12.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.542+0000 7f72e35bc640 10 monclient: _renew_subs 2024-03-29T11:17:12.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.542+0000 7f72e35bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:17:12.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.542+0000 7f72e35bc640 1 -- 172.21.15.18:0/3386163589 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f72dc133900 con 0x7f72dc0b0770 2024-03-29T11:17:12.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.542+0000 7f72e35bc640 1 librados: init done 2024-03-29T11:17:12.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.543+0000 7f72d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:12.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.543+0000 7f72d1ffb640 1 -- 172.21.15.18:0/3386163589 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f72cc004ac0 con 0x7f72dc0b0770 2024-03-29T11:17:12.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.543+0000 7f72d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:12.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.543+0000 7f72d1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:12.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.543+0000 7f72d1ffb640 10 monclient: dump: 2024-03-29T11:17:12.525 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:17:12.525 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:12.525 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:12.525 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:12.525 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:12.525 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:12.525 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:17:12.525 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:17:12.525 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:17:12.525 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:12.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.543+0000 7f72d37fe640 4 set_mon_vals no callback set 2024-03-29T11:17:12.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.544+0000 7f72d1ffb640 1 -- 172.21.15.18:0/3386163589 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f72cc01aa90 con 0x7f72dc0b0770 2024-03-29T11:17:12.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.545+0000 7f72e35bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:12.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.545+0000 7f72e35bc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:12.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.545+0000 7f72e35bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:17:12.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.545+0000 7f72e35bc640 1 -- 172.21.15.18:0/3386163589 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72bc005080 con 0x7f72dc0b0770 2024-03-29T11:17:12.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.545+0000 7f72e1358640 10 monclient: get_auth_request con 0x7f72c006dd10 auth_method 0 2024-03-29T11:17:12.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.545+0000 7f72d1ffb640 1 -- 172.21.15.18:0/3386163589 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f72cc084b10 con 0x7f72dc0b0770 2024-03-29T11:17:12.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.545+0000 7f72d1ffb640 10 client.4571.objecter ms_dispatch 0x7f72dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:12.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.545+0000 7f72d1ffb640 3 client.4571.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:12.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.545+0000 7f72d1ffb640 3 client.4571.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:12.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.546+0000 7f72d1ffb640 20 client.4571.objecter dump_active .. 0 homeless 2024-03-29T11:17:12.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.546+0000 7f72d1ffb640 10 client.4571.objecter ms_handle_connect 0x7f72c006dd10 2024-03-29T11:17:12.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.552+0000 7f72d1ffb640 1 -- 172.21.15.18:0/3386163589 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f72cc0503f0 con 0x7f72dc0b0770 2024-03-29T11:17:12.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.552+0000 7f72d1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:12.530 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.552+0000 7f72d1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:12.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.861+0000 7f72e35bc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:12.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.861+0000 7f72e35bc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:12.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.861+0000 7f72e35bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:17:12.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.861+0000 7f72e35bc640 1 -- 172.21.15.18:0/3386163589 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72bc005330 con 0x7f72dc0b0770 2024-03-29T11:17:12.839 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:17:12.839 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:12.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.862+0000 7f72d1ffb640 1 -- 172.21.15.18:0/3386163589 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f72cc052320 con 0x7f72dc0b0770 2024-03-29T11:17:12.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.862+0000 7f72d1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:12.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.862+0000 7f72d1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:12.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.865+0000 7f72c77fe640 10 librados: watch_flush enter 2024-03-29T11:17:12.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.865+0000 7f72c77fe640 10 librados: watch_flush exit 2024-03-29T11:17:12.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.865+0000 7f72c77fe640 20 client.4571.objecter shutdown clearing up homeless session... 2024-03-29T11:17:12.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.865+0000 7f72c77fe640 10 client.4571.objecter successfully canceled tick 2024-03-29T11:17:12.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.865+0000 7f72c77fe640 1 -- 172.21.15.18:0/3386163589 >> v1:172.21.15.70:6816/3101871562 conn(0x7f72c006dd10 legacy=0x7f72c00701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:12.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.865+0000 7f72c77fe640 10 monclient: shutdown 2024-03-29T11:17:12.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.865+0000 7f72c77fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:12.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.865+0000 7f72c77fe640 1 -- 172.21.15.18:0/3386163589 >> v1:172.21.15.18:6790/0 conn(0x7f72dc0b0770 legacy=0x7f72dc1d33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:12.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.865+0000 7f72c77fe640 1 -- 172.21.15.18:0/3386163589 shutdown_connections 2024-03-29T11:17:12.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.865+0000 7f72c77fe640 1 -- 172.21.15.18:0/3386163589 >> 172.21.15.18:0/3386163589 conn(0x7f72dc06b660 msgr2=0x7f72dc0a9db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:12.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.865+0000 7f72c77fe640 1 -- 172.21.15.18:0/3386163589 shutdown_connections 2024-03-29T11:17:12.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.865+0000 7f72c77fe640 1 -- 172.21.15.18:0/3386163589 wait complete. 2024-03-29T11:17:12.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:12.866+0000 7f72c77fe640 1 librados: shutdown 2024-03-29T11:17:12.859 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:19.667 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:19.811 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.834+0000 7f0169ee0640 10 monclient: get_monmap_and_config 2024-03-29T11:17:19.812 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.834+0000 7f0169ee0640 10 monclient: build_initial_monmap 2024-03-29T11:17:19.812 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.834+0000 7f0169ee0640 10 monclient: monmap: 2024-03-29T11:17:19.812 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:17:19.812 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:19.812 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:17:19.835519+0000 2024-03-29T11:17:19.812 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:17:19.835519+0000 2024-03-29T11:17:19.812 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:19.812 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:19.812 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:17:19.812 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:17:19.812 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:17:19.813 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:19.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.837+0000 7f0169ee0640 1 Processor -- start 2024-03-29T11:17:19.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.837+0000 7f0169ee0640 1 -- start start 2024-03-29T11:17:19.814 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.837+0000 7f0169ee0640 10 monclient: init 2024-03-29T11:17:19.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.837+0000 7f0169ee0640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:19.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.837+0000 7f0169ee0640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:17:19.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.838+0000 7f0169ee0640 10 monclient(hunting): picked mon.noname-b con 0x7f01640b1f60 addr v1:172.21.15.18:6790/0 2024-03-29T11:17:19.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.838+0000 7f0169ee0640 10 monclient(hunting): picked mon.noname-a con 0x7f0164071eb0 addr v1:172.21.15.18:6789/0 2024-03-29T11:17:19.815 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.838+0000 7f0169ee0640 10 monclient(hunting): picked mon.noname-c con 0x7f0164143d10 addr v1:172.21.15.70:6789/0 2024-03-29T11:17:19.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.838+0000 7f0169ee0640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f016406ef80 con 0x7f0164071eb0 2024-03-29T11:17:19.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.838+0000 7f0169ee0640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f016406f260 con 0x7f0164143d10 2024-03-29T11:17:19.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.838+0000 7f0169ee0640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01640b40c0 con 0x7f01640b1f60 2024-03-29T11:17:19.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.838+0000 7f0169ee0640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:19.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.838+0000 7f0169ee0640 10 monclient(hunting): authenticate will time out at 1266.109998s 2024-03-29T11:17:19.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.838+0000 7f01637fe640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f01640b1f60 0x7f0164143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:52354/0 (socket says 172.21.15.18:52354) 2024-03-29T11:17:19.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.838+0000 7f01637fe640 1 -- 172.21.15.18:0/852205381 learned_addr learned my addr 172.21.15.18:0/852205381 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:17:19.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.839+0000 7f01627fc640 1 -- 172.21.15.18:0/852205381 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2745278461 0 0) 0x7f01640b40c0 con 0x7f01640b1f60 2024-03-29T11:17:19.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.839+0000 7f01627fc640 10 monclient(hunting): _negotiate 2024-03-29T11:17:19.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.839+0000 7f01627fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:19.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.839+0000 7f01627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:19.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.839+0000 7f01627fc640 10 monclient(hunting): my global_id is 4574 2024-03-29T11:17:19.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.839+0000 7f01627fc640 1 -- 172.21.15.18:0/852205381 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f014c003520 con 0x7f01640b1f60 2024-03-29T11:17:19.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.839+0000 7f01627fc640 1 -- 172.21.15.18:0/852205381 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 187267702 0 0) 0x7f014c003520 con 0x7f01640b1f60 2024-03-29T11:17:19.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.839+0000 7f01627fc640 1 -- 172.21.15.18:0/852205381 >> v1:172.21.15.70:6789/0 conn(0x7f0164143d10 legacy=0x7f0164146b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:17:19.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.841+0000 7f01627fc640 1 -- 172.21.15.18:0/852205381 >> v1:172.21.15.18:6789/0 conn(0x7f0164071eb0 legacy=0x7f0164146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:17:19.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.841+0000 7f01627fc640 10 monclient: _finish_hunting 0 2024-03-29T11:17:19.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.841+0000 7f01627fc640 1 monclient: found mon.noname-b 2024-03-29T11:17:19.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.841+0000 7f01627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:19.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.841+0000 7f01627fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:17:19.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.841+0000 7f01627fc640 1 -- 172.21.15.18:0/852205381 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0164155d70 con 0x7f01640b1f60 2024-03-29T11:17:19.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.841+0000 7f01627fc640 10 monclient: _finish_auth 0 2024-03-29T11:17:19.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.841+0000 7f01627fc640 10 monclient: _check_auth_tickets 2024-03-29T11:17:19.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.841+0000 7f01627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:19.820 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.841+0000 7f01627fc640 1 -- 172.21.15.18:0/852205381 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f0158001690 con 0x7f01640b1f60 2024-03-29T11:17:19.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.841+0000 7f01627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:19.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.841+0000 7f01627fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:17:19.821 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.841+0000 7f01627fc640 10 monclient: dump: 2024-03-29T11:17:19.821 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:17:19.821 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:19.821 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:19.821 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:19.821 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:19.821 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:19.821 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:17:19.821 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:17:19.821 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:17:19.821 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:19.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.842+0000 7f01627fc640 1 -- 172.21.15.18:0/852205381 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0158001860 con 0x7f01640b1f60 2024-03-29T11:17:19.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.842+0000 7f01627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:19.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.842+0000 7f01627fc640 1 -- 172.21.15.18:0/852205381 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f0158001cb0 con 0x7f01640b1f60 2024-03-29T11:17:19.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.842+0000 7f01627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:19.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.842+0000 7f01627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:19.822 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.842+0000 7f01627fc640 10 monclient: dump: 2024-03-29T11:17:19.822 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:17:19.822 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:19.822 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:19.822 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:19.822 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:19.822 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:19.822 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:17:19.822 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:17:19.823 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:17:19.823 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:19.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.842+0000 7f0169ee0640 5 monclient: authenticate success, global_id 4574 2024-03-29T11:17:19.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.842+0000 7f0169ee0640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:19.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.842+0000 7f0169ee0640 4 set_mon_vals no callback set 2024-03-29T11:17:19.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.844+0000 7f0169ee0640 10 monclient: shutdown 2024-03-29T11:17:19.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.844+0000 7f0169ee0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:19.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.844+0000 7f0169ee0640 1 -- 172.21.15.18:0/852205381 >> v1:172.21.15.18:6790/0 conn(0x7f01640b1f60 legacy=0x7f0164143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:19.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.845+0000 7f0169ee0640 1 -- 172.21.15.18:0/852205381 shutdown_connections 2024-03-29T11:17:19.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.845+0000 7f0169ee0640 1 -- 172.21.15.18:0/852205381 >> 172.21.15.18:0/852205381 conn(0x7f0164066a00 msgr2=0x7f0164066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:19.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.846+0000 7f0169ee0640 1 -- 172.21.15.18:0/852205381 shutdown_connections 2024-03-29T11:17:19.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.846+0000 7f0169ee0640 1 -- 172.21.15.18:0/852205381 wait complete. 2024-03-29T11:17:19.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.847+0000 7f0169ee0640 10 monclient: build_initial_monmap 2024-03-29T11:17:19.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.847+0000 7f0169ee0640 10 monclient: monmap: 2024-03-29T11:17:19.827 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:17:19.827 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:19.827 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:17:19.827 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:17:19.827 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:19.827 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:19.827 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:17:19.827 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:17:19.827 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:17:19.827 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:19.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.847+0000 7f0169ee0640 1 librados: starting msgr at 2024-03-29T11:17:19.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.847+0000 7f0169ee0640 1 librados: starting objecter 2024-03-29T11:17:19.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.847+0000 7f0169ee0640 1 Processor -- start 2024-03-29T11:17:19.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.848+0000 7f0169ee0640 1 -- start start 2024-03-29T11:17:19.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.848+0000 7f0169ee0640 1 librados: setting wanted keys 2024-03-29T11:17:19.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.848+0000 7f0169ee0640 1 librados: calling monclient init 2024-03-29T11:17:19.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.848+0000 7f0169ee0640 10 monclient: init 2024-03-29T11:17:19.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.848+0000 7f0169ee0640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:19.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.848+0000 7f0169ee0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:17:19.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.848+0000 7f0169ee0640 10 monclient(hunting): picked mon.noname-b con 0x7f0164143d10 addr v1:172.21.15.70:6789/0 2024-03-29T11:17:19.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.848+0000 7f0169ee0640 10 monclient(hunting): picked mon.noname-a con 0x7f0164071320 addr v1:172.21.15.18:6789/0 2024-03-29T11:17:19.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.848+0000 7f0169ee0640 10 monclient(hunting): picked mon.noname-c con 0x7f0164131ee0 addr v1:172.21.15.18:6790/0 2024-03-29T11:17:19.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.848+0000 7f0169ee0640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0164141e50 con 0x7f0164071320 2024-03-29T11:17:19.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.848+0000 7f0169ee0640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01641420a0 con 0x7f0164143d10 2024-03-29T11:17:19.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.848+0000 7f0169ee0640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0164142380 con 0x7f0164131ee0 2024-03-29T11:17:19.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.848+0000 7f0169ee0640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:19.828 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.848+0000 7f0169ee0640 10 monclient(hunting): authenticate will time out at 1266.120725s 2024-03-29T11:17:19.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.849+0000 7f0163fff640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f0164131ee0 0x7f01641e6f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:52580/0 (socket says 172.21.15.18:52580) 2024-03-29T11:17:19.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.849+0000 7f0163fff640 1 -- 172.21.15.18:0/3621512461 learned_addr learned my addr 172.21.15.18:0/3621512461 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:17:19.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.849+0000 7f014bfff640 10 client.?.objecter ms_handle_connect 0x7f0164131ee0 2024-03-29T11:17:19.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.849+0000 7f014bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:19.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.849+0000 7f014bfff640 1 -- 172.21.15.18:0/3621512461 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3794150019 0 0) 0x7f0164142380 con 0x7f0164131ee0 2024-03-29T11:17:19.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.849+0000 7f014bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:17:19.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.849+0000 7f014bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:19.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.849+0000 7f014bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:19.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.849+0000 7f014bfff640 10 monclient(hunting): my global_id is 4577 2024-03-29T11:17:19.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.849+0000 7f014bfff640 1 -- 172.21.15.18:0/3621512461 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0150004eb0 con 0x7f0164131ee0 2024-03-29T11:17:19.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.850+0000 7f014bfff640 1 -- 172.21.15.18:0/3621512461 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 891588277 0 0) 0x7f0150004eb0 con 0x7f0164131ee0 2024-03-29T11:17:19.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.850+0000 7f014bfff640 1 -- 172.21.15.18:0/3621512461 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0164142380 con 0x7f0164131ee0 2024-03-29T11:17:19.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.850+0000 7f014bfff640 1 -- 172.21.15.18:0/3621512461 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f0154001750 con 0x7f0164131ee0 2024-03-29T11:17:19.831 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.850+0000 7f014bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:19.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.850+0000 7f014bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:19.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.850+0000 7f014bfff640 10 monclient(hunting): dump: 2024-03-29T11:17:19.832 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:17:19.832 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:19.832 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:19.832 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:19.832 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:19.832 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:19.832 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:17:19.832 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:17:19.832 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:17:19.832 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:19.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.850+0000 7f014bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:19.832 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.851+0000 7f014bfff640 1 -- 172.21.15.18:0/3621512461 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2884653136 0 0) 0x7f0164142380 con 0x7f0164131ee0 2024-03-29T11:17:19.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.851+0000 7f014bfff640 1 -- 172.21.15.18:0/3621512461 >> v1:172.21.15.70:6789/0 conn(0x7f0164143d10 legacy=0x7f0164142a80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:17:19.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.853+0000 7f014bfff640 1 -- 172.21.15.18:0/3621512461 >> v1:172.21.15.18:6789/0 conn(0x7f0164071320 legacy=0x7f01641342b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:19.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.853+0000 7f014bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:17:19.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.853+0000 7f014bfff640 1 monclient: found mon.c 2024-03-29T11:17:19.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.853+0000 7f014bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:19.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.853+0000 7f014bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:17:19.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.853+0000 7f014bfff640 1 -- 172.21.15.18:0/3621512461 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0164142660 con 0x7f0164131ee0 2024-03-29T11:17:19.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.853+0000 7f014bfff640 10 monclient: _finish_auth 0 2024-03-29T11:17:19.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.853+0000 7f014bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:17:19.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.853+0000 7f014bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:19.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.853+0000 7f014bfff640 10 client.?.objecter ms_handle_connect 0x7f0164071320 2024-03-29T11:17:19.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.853+0000 7f014bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:19.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.853+0000 7f0169ee0640 5 monclient: authenticate success, global_id 4577 2024-03-29T11:17:19.833 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.853+0000 7f0169ee0640 10 monclient: _renew_subs 2024-03-29T11:17:19.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.853+0000 7f0169ee0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:17:19.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.853+0000 7f0169ee0640 1 -- 172.21.15.18:0/3621512461 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01641414e0 con 0x7f0164131ee0 2024-03-29T11:17:19.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.853+0000 7f0169ee0640 10 client.4577.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:19.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.853+0000 7f0169ee0640 10 monclient: _renew_subs 2024-03-29T11:17:19.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.853+0000 7f0169ee0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:17:19.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.853+0000 7f0169ee0640 1 -- 172.21.15.18:0/3621512461 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f01641536e0 con 0x7f0164131ee0 2024-03-29T11:17:19.834 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.853+0000 7f0169ee0640 1 librados: init done 2024-03-29T11:17:19.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.855+0000 7f014bfff640 1 -- 172.21.15.18:0/3621512461 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0154001d20 con 0x7f0164131ee0 2024-03-29T11:17:19.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.855+0000 7f014bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:19.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.855+0000 7f014bfff640 1 -- 172.21.15.18:0/3621512461 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f0154004a00 con 0x7f0164131ee0 2024-03-29T11:17:19.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.855+0000 7f014bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:19.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.855+0000 7f014bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:19.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.855+0000 7f014bfff640 10 monclient: dump: 2024-03-29T11:17:19.836 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:17:19.836 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:19.836 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:19.836 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:19.836 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:19.836 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:19.836 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:17:19.836 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:17:19.837 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:17:19.837 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:19.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.855+0000 7f0161ffb640 4 set_mon_vals no callback set 2024-03-29T11:17:19.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.855+0000 7f0169ee0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:19.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.856+0000 7f0169ee0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:19.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.856+0000 7f0169ee0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:17:19.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.856+0000 7f0169ee0640 1 -- 172.21.15.18:0/3621512461 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0144005080 con 0x7f0164131ee0 2024-03-29T11:17:19.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.857+0000 7f014bfff640 1 -- 172.21.15.18:0/3621512461 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f015401a9d0 con 0x7f0164131ee0 2024-03-29T11:17:19.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.858+0000 7f014bfff640 1 -- 172.21.15.18:0/3621512461 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f01540849e0 con 0x7f0164131ee0 2024-03-29T11:17:19.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.858+0000 7f014bfff640 10 client.4577.objecter ms_dispatch 0x7f0164000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:19.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.858+0000 7f014bfff640 3 client.4577.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:19.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.858+0000 7f014bfff640 3 client.4577.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:19.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.858+0000 7f014bfff640 20 client.4577.objecter dump_active .. 0 homeless 2024-03-29T11:17:19.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.861+0000 7f01637fe640 10 monclient: get_auth_request con 0x7f015006db10 auth_method 0 2024-03-29T11:17:19.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.862+0000 7f014bfff640 10 client.4577.objecter ms_handle_connect 0x7f015006db10 2024-03-29T11:17:19.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.864+0000 7f014bfff640 1 -- 172.21.15.18:0/3621512461 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f01540500f0 con 0x7f0164131ee0 2024-03-29T11:17:19.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.864+0000 7f014bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:19.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:19.864+0000 7f014bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:20.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:20.203+0000 7f0169ee0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:20.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:20.203+0000 7f0169ee0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:20.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:20.203+0000 7f0169ee0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:17:20.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:20.203+0000 7f0169ee0640 1 -- 172.21.15.18:0/3621512461 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0144005330 con 0x7f0164131ee0 2024-03-29T11:17:20.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:20.206+0000 7f014bfff640 1 -- 172.21.15.18:0/3621512461 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f01540521f0 con 0x7f0164131ee0 2024-03-29T11:17:20.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:20.206+0000 7f014bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:20.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:20.206+0000 7f014bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:20.184 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:17:20.184 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:20.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:20.209+0000 7f0169ee0640 10 librados: watch_flush enter 2024-03-29T11:17:20.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:20.209+0000 7f0169ee0640 10 librados: watch_flush exit 2024-03-29T11:17:20.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:20.209+0000 7f0169ee0640 20 client.4577.objecter shutdown clearing up homeless session... 2024-03-29T11:17:20.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:20.209+0000 7f0169ee0640 10 client.4577.objecter successfully canceled tick 2024-03-29T11:17:20.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:20.210+0000 7f0169ee0640 1 -- 172.21.15.18:0/3621512461 >> v1:172.21.15.70:6816/3101871562 conn(0x7f015006db10 legacy=0x7f015006ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:20.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:20.210+0000 7f0169ee0640 10 monclient: shutdown 2024-03-29T11:17:20.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:20.210+0000 7f0169ee0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:20.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:20.210+0000 7f0169ee0640 1 -- 172.21.15.18:0/3621512461 >> v1:172.21.15.18:6790/0 conn(0x7f0164131ee0 legacy=0x7f01641e6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:20.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:20.210+0000 7f0169ee0640 1 -- 172.21.15.18:0/3621512461 shutdown_connections 2024-03-29T11:17:20.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:20.210+0000 7f0169ee0640 1 -- 172.21.15.18:0/3621512461 >> 172.21.15.18:0/3621512461 conn(0x7f0164066a00 msgr2=0x7f0164146980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:20.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:20.212+0000 7f0169ee0640 1 -- 172.21.15.18:0/3621512461 shutdown_connections 2024-03-29T11:17:20.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:20.212+0000 7f0169ee0640 1 -- 172.21.15.18:0/3621512461 wait complete. 2024-03-29T11:17:20.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:20.212+0000 7f0169ee0640 1 librados: shutdown 2024-03-29T11:17:20.206 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:27.014 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:27.156 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.179+0000 7f1acd077640 10 monclient: get_monmap_and_config 2024-03-29T11:17:27.156 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.179+0000 7f1acd077640 10 monclient: build_initial_monmap 2024-03-29T11:17:27.156 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.179+0000 7f1acd077640 10 monclient: monmap: 2024-03-29T11:17:27.156 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:17:27.156 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:27.156 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:17:27.180106+0000 2024-03-29T11:17:27.157 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:17:27.180106+0000 2024-03-29T11:17:27.157 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:27.157 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:27.157 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:17:27.157 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:17:27.157 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:17:27.157 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:27.160 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.183+0000 7f1acd077640 1 Processor -- start 2024-03-29T11:17:27.163 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.183+0000 7f1acd077640 1 -- start start 2024-03-29T11:17:27.163 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.183+0000 7f1acd077640 10 monclient: init 2024-03-29T11:17:27.163 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.184+0000 7f1acd077640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:27.163 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.184+0000 7f1acd077640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:17:27.163 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.184+0000 7f1acd077640 10 monclient(hunting): picked mon.noname-a con 0x7f1ac0038c60 addr v1:172.21.15.18:6789/0 2024-03-29T11:17:27.163 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.184+0000 7f1acd077640 10 monclient(hunting): picked mon.noname-b con 0x7f1ac00324b0 addr v1:172.21.15.18:6790/0 2024-03-29T11:17:27.163 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.184+0000 7f1acd077640 10 monclient(hunting): picked mon.noname-c con 0x7f1ac00d11e0 addr v1:172.21.15.70:6789/0 2024-03-29T11:17:27.163 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.184+0000 7f1acd077640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac00caf80 con 0x7f1ac0038c60 2024-03-29T11:17:27.163 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.184+0000 7f1acd077640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac0039850 con 0x7f1ac00d11e0 2024-03-29T11:17:27.163 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.184+0000 7f1acd077640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac00c78d0 con 0x7f1ac00324b0 2024-03-29T11:17:27.163 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.184+0000 7f1acd077640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:27.163 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.184+0000 7f1acd077640 10 monclient(hunting): authenticate will time out at 1273.456748s 2024-03-29T11:17:27.163 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.185+0000 7f1ac77fe640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f1ac00324b0 0x7f1ac00cb4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:41586/0 (socket says 172.21.15.18:41586) 2024-03-29T11:17:27.164 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.185+0000 7f1ac77fe640 1 -- 172.21.15.18:0/3172033067 learned_addr learned my addr 172.21.15.18:0/3172033067 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:17:27.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.187+0000 7f1ac6ffd640 1 -- 172.21.15.18:0/3172033067 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3577303673 0 0) 0x7f1ac00c78d0 con 0x7f1ac00324b0 2024-03-29T11:17:27.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.187+0000 7f1ac6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:17:27.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.187+0000 7f1ac6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:27.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.187+0000 7f1ac6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:27.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.187+0000 7f1ac6ffd640 10 monclient(hunting): my global_id is 4580 2024-03-29T11:17:27.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.187+0000 7f1ac6ffd640 1 -- 172.21.15.18:0/3172033067 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ab8003520 con 0x7f1ac00324b0 2024-03-29T11:17:27.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.187+0000 7f1ac6ffd640 1 -- 172.21.15.18:0/3172033067 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1222766001 0 0) 0x7f1ab8003520 con 0x7f1ac00324b0 2024-03-29T11:17:27.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.187+0000 7f1ac6ffd640 1 -- 172.21.15.18:0/3172033067 >> v1:172.21.15.70:6789/0 conn(0x7f1ac00d11e0 legacy=0x7f1ac00d35d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:17:27.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.190+0000 7f1ac6ffd640 1 -- 172.21.15.18:0/3172033067 >> v1:172.21.15.18:6789/0 conn(0x7f1ac0038c60 legacy=0x7f1ac00c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:27.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.190+0000 7f1ac6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:17:27.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.191+0000 7f1ac6ffd640 1 monclient: found mon.noname-b 2024-03-29T11:17:27.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.191+0000 7f1ac6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:27.168 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.191+0000 7f1ac6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:17:27.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.191+0000 7f1ac6ffd640 1 -- 172.21.15.18:0/3172033067 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac00c7a60 con 0x7f1ac00324b0 2024-03-29T11:17:27.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.191+0000 7f1ac6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:17:27.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.191+0000 7f1ac6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:17:27.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.191+0000 7f1ac6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:27.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.191+0000 7f1ac6ffd640 1 -- 172.21.15.18:0/3172033067 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f1abc001690 con 0x7f1ac00324b0 2024-03-29T11:17:27.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.191+0000 7f1ac6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:27.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.191+0000 7f1ac6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:17:27.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.191+0000 7f1ac6ffd640 10 monclient: dump: 2024-03-29T11:17:27.169 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:17:27.169 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:27.169 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:27.169 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:27.169 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:27.169 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:27.170 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:17:27.170 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:17:27.170 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:17:27.170 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:27.170 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.191+0000 7f1acd077640 5 monclient: authenticate success, global_id 4580 2024-03-29T11:17:27.170 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.191+0000 7f1acd077640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:27.170 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.191+0000 7f1ac6ffd640 1 -- 172.21.15.18:0/3172033067 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1abc001920 con 0x7f1ac00324b0 2024-03-29T11:17:27.170 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.191+0000 7f1ac6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:27.170 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.191+0000 7f1acd077640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:27.170 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.191+0000 7f1acd077640 4 set_mon_vals no callback set 2024-03-29T11:17:27.170 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.191+0000 7f1ac6ffd640 1 -- 172.21.15.18:0/3172033067 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f1abc001cb0 con 0x7f1ac00324b0 2024-03-29T11:17:27.170 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.191+0000 7f1ac6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:27.170 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.191+0000 7f1ac6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:27.170 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.191+0000 7f1ac6ffd640 10 monclient: dump: 2024-03-29T11:17:27.171 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:17:27.171 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:27.171 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:27.171 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:27.171 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:27.171 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:27.171 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:17:27.171 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:17:27.171 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:17:27.171 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:27.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.192+0000 7f1acd077640 10 monclient: shutdown 2024-03-29T11:17:27.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.192+0000 7f1acd077640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:27.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.192+0000 7f1acd077640 1 -- 172.21.15.18:0/3172033067 >> v1:172.21.15.18:6790/0 conn(0x7f1ac00324b0 legacy=0x7f1ac00cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:27.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.193+0000 7f1acd077640 1 -- 172.21.15.18:0/3172033067 shutdown_connections 2024-03-29T11:17:27.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.193+0000 7f1acd077640 1 -- 172.21.15.18:0/3172033067 >> 172.21.15.18:0/3172033067 conn(0x7f1ac0005c20 msgr2=0x7f1ac0006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:27.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.193+0000 7f1acd077640 1 -- 172.21.15.18:0/3172033067 shutdown_connections 2024-03-29T11:17:27.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.193+0000 7f1acd077640 1 -- 172.21.15.18:0/3172033067 wait complete. 2024-03-29T11:17:27.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.194+0000 7f1acd077640 10 monclient: build_initial_monmap 2024-03-29T11:17:27.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.194+0000 7f1acd077640 10 monclient: monmap: 2024-03-29T11:17:27.173 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:17:27.173 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:27.173 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:17:27.173 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:17:27.174 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:27.174 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:27.174 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:17:27.174 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:17:27.174 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:17:27.174 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:27.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.194+0000 7f1acd077640 1 librados: starting msgr at 2024-03-29T11:17:27.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.194+0000 7f1acd077640 1 librados: starting objecter 2024-03-29T11:17:27.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.195+0000 7f1acd077640 1 Processor -- start 2024-03-29T11:17:27.175 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.195+0000 7f1acd077640 1 -- start start 2024-03-29T11:17:27.175 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.195+0000 7f1acd077640 1 librados: setting wanted keys 2024-03-29T11:17:27.175 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.195+0000 7f1acd077640 1 librados: calling monclient init 2024-03-29T11:17:27.175 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.195+0000 7f1acd077640 10 monclient: init 2024-03-29T11:17:27.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.196+0000 7f1acd077640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:27.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.196+0000 7f1acd077640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:17:27.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.196+0000 7f1acd077640 10 monclient(hunting): picked mon.noname-a con 0x7f1ac00324b0 addr v1:172.21.15.18:6789/0 2024-03-29T11:17:27.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.196+0000 7f1acd077640 10 monclient(hunting): picked mon.noname-c con 0x7f1ac0038c60 addr v1:172.21.15.18:6790/0 2024-03-29T11:17:27.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.196+0000 7f1acd077640 10 monclient(hunting): picked mon.noname-b con 0x7f1ac00d11e0 addr v1:172.21.15.70:6789/0 2024-03-29T11:17:27.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.196+0000 7f1acd077640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac00c26b0 con 0x7f1ac00324b0 2024-03-29T11:17:27.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.196+0000 7f1acd077640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac00c28e0 con 0x7f1ac00d11e0 2024-03-29T11:17:27.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.196+0000 7f1acd077640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac0036a20 con 0x7f1ac0038c60 2024-03-29T11:17:27.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.196+0000 7f1acd077640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:27.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.196+0000 7f1acd077640 10 monclient(hunting): authenticate will time out at 1273.468304s 2024-03-29T11:17:27.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.199+0000 7f1ac77fe640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f1ac0038c60 0x7f1ac00c3ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:45406/0 (socket says 172.21.15.18:45406) 2024-03-29T11:17:27.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.199+0000 7f1ac77fe640 1 -- 172.21.15.18:0/675592271 learned_addr learned my addr 172.21.15.18:0/675592271 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:17:27.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.200+0000 7f1ac4ff9640 10 client.?.objecter ms_handle_connect 0x7f1ac0038c60 2024-03-29T11:17:27.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.200+0000 7f1ac4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:27.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.200+0000 7f1ac4ff9640 10 client.?.objecter ms_handle_connect 0x7f1ac00d11e0 2024-03-29T11:17:27.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.200+0000 7f1ac4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:27.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.200+0000 7f1ac4ff9640 10 client.?.objecter ms_handle_connect 0x7f1ac00324b0 2024-03-29T11:17:27.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.200+0000 7f1ac4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:27.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.200+0000 7f1ac4ff9640 1 -- 172.21.15.18:0/675592271 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2996341672 0 0) 0x7f1ac0036a20 con 0x7f1ac0038c60 2024-03-29T11:17:27.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.200+0000 7f1ac4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:17:27.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.200+0000 7f1ac4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:27.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.200+0000 7f1ac4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:27.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.200+0000 7f1ac4ff9640 10 monclient(hunting): my global_id is 4583 2024-03-29T11:17:27.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.200+0000 7f1ac4ff9640 1 -- 172.21.15.18:0/675592271 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1aac004eb0 con 0x7f1ac0038c60 2024-03-29T11:17:27.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.200+0000 7f1ac4ff9640 1 -- 172.21.15.18:0/675592271 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1432032999 0 0) 0x7f1ac00c28e0 con 0x7f1ac00d11e0 2024-03-29T11:17:27.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.200+0000 7f1ac4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:17:27.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.200+0000 7f1ac4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:27.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.200+0000 7f1ac4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:27.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.200+0000 7f1ac4ff9640 10 monclient(hunting): my global_id is 4465 2024-03-29T11:17:27.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.200+0000 7f1ac4ff9640 1 -- 172.21.15.18:0/675592271 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ac0036a20 con 0x7f1ac00d11e0 2024-03-29T11:17:27.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.200+0000 7f1ac4ff9640 1 -- 172.21.15.18:0/675592271 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 757223045 0 0) 0x7f1ac00c26b0 con 0x7f1ac00324b0 2024-03-29T11:17:27.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.200+0000 7f1ac4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:17:27.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.200+0000 7f1ac4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:27.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.200+0000 7f1ac4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:27.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.200+0000 7f1ac4ff9640 10 monclient(hunting): my global_id is 4497 2024-03-29T11:17:27.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.201+0000 7f1ac4ff9640 1 -- 172.21.15.18:0/675592271 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ac00c28e0 con 0x7f1ac00324b0 2024-03-29T11:17:27.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.201+0000 7f1ac4ff9640 1 -- 172.21.15.18:0/675592271 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 966669291 0 0) 0x7f1ac0036a20 con 0x7f1ac00d11e0 2024-03-29T11:17:27.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.201+0000 7f1ac4ff9640 1 -- 172.21.15.18:0/675592271 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ac00c26b0 con 0x7f1ac00d11e0 2024-03-29T11:17:27.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.201+0000 7f1ac4ff9640 1 -- 172.21.15.18:0/675592271 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 331869742 0 0) 0x7f1aac004eb0 con 0x7f1ac0038c60 2024-03-29T11:17:27.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.201+0000 7f1ac4ff9640 1 -- 172.21.15.18:0/675592271 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ac0036a20 con 0x7f1ac0038c60 2024-03-29T11:17:27.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.201+0000 7f1ac4ff9640 1 -- 172.21.15.18:0/675592271 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f1ab4001710 con 0x7f1ac00d11e0 2024-03-29T11:17:27.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.201+0000 7f1ac4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:27.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.201+0000 7f1ac4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:17:27.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.201+0000 7f1ac4ff9640 10 monclient(hunting): dump: 2024-03-29T11:17:27.187 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:17:27.187 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:27.187 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:27.187 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:27.187 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:27.187 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:27.187 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:17:27.187 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:17:27.188 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:17:27.188 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:27.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.201+0000 7f1ac4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:27.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.201+0000 7f1ac4ff9640 1 -- 172.21.15.18:0/675592271 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 633608367 0 0) 0x7f1ac00c28e0 con 0x7f1ac00324b0 2024-03-29T11:17:27.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.201+0000 7f1ac4ff9640 1 -- 172.21.15.18:0/675592271 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1aac004eb0 con 0x7f1ac00324b0 2024-03-29T11:17:27.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.201+0000 7f1ac4ff9640 1 -- 172.21.15.18:0/675592271 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f1abc001590 con 0x7f1ac0038c60 2024-03-29T11:17:27.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.201+0000 7f1ac4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:27.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.201+0000 7f1ac4ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:27.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.201+0000 7f1ac4ff9640 10 monclient(hunting): dump: 2024-03-29T11:17:27.188 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:17:27.188 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:27.188 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:27.188 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:27.188 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:27.189 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:27.189 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:17:27.189 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:17:27.189 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:17:27.189 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:27.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.201+0000 7f1ac4ff9640 1 -- 172.21.15.18:0/675592271 <== mon.1 v1:172.21.15.70:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 808997155 0 0) 0x7f1ac00c26b0 con 0x7f1ac00d11e0 2024-03-29T11:17:27.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.201+0000 7f1ac4ff9640 1 -- 172.21.15.18:0/675592271 >> v1:172.21.15.18:6790/0 conn(0x7f1ac0038c60 legacy=0x7f1ac00c3ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:27.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.201+0000 7f1ac4ff9640 1 -- 172.21.15.18:0/675592271 >> v1:172.21.15.18:6789/0 conn(0x7f1ac00324b0 legacy=0x7f1ac00c33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:27.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.201+0000 7f1ac4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:17:27.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.201+0000 7f1ac4ff9640 1 monclient: found mon.b 2024-03-29T11:17:27.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.201+0000 7f1ac4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:27.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.201+0000 7f1ac4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:17:27.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.201+0000 7f1ac4ff9640 1 -- 172.21.15.18:0/675592271 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac00353c0 con 0x7f1ac00d11e0 2024-03-29T11:17:27.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.201+0000 7f1ac4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:17:27.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.201+0000 7f1ac4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:17:27.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.201+0000 7f1ac4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:27.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.201+0000 7f1ac4ff9640 1 -- 172.21.15.18:0/675592271 <== mon.1 v1:172.21.15.70:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ab4001870 con 0x7f1ac00d11e0 2024-03-29T11:17:27.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.201+0000 7f1ac4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:27.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.201+0000 7f1ac67fc640 4 set_mon_vals no callback set 2024-03-29T11:17:27.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.202+0000 7f1acd077640 5 monclient: authenticate success, global_id 4465 2024-03-29T11:17:27.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.202+0000 7f1acd077640 10 monclient: _renew_subs 2024-03-29T11:17:27.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.202+0000 7f1acd077640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:17:27.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.202+0000 7f1acd077640 1 -- 172.21.15.18:0/675592271 --> v1:172.21.15.70:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ac00c9990 con 0x7f1ac00d11e0 2024-03-29T11:17:27.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.202+0000 7f1ac4ff9640 1 -- 172.21.15.18:0/675592271 <== mon.1 v1:172.21.15.70:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f1ab40035f0 con 0x7f1ac00d11e0 2024-03-29T11:17:27.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.202+0000 7f1ac4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:27.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.202+0000 7f1ac4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:17:27.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.202+0000 7f1ac4ff9640 10 monclient: dump: 2024-03-29T11:17:27.190 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:17:27.190 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:27.191 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:27.191 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:27.191 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:27.191 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:27.191 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:17:27.191 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:17:27.191 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:17:27.191 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:27.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.202+0000 7f1acd077640 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:27.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.202+0000 7f1acd077640 10 monclient: _renew_subs 2024-03-29T11:17:27.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.202+0000 7f1acd077640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:17:27.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.202+0000 7f1acd077640 1 -- 172.21.15.18:0/675592271 --> v1:172.21.15.70:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ac00363d0 con 0x7f1ac00d11e0 2024-03-29T11:17:27.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.202+0000 7f1acd077640 1 librados: init done 2024-03-29T11:17:27.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.203+0000 7f1acd077640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:27.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.203+0000 7f1acd077640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:27.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.203+0000 7f1acd077640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:17:27.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.203+0000 7f1acd077640 1 -- 172.21.15.18:0/675592271 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ac0007b90 con 0x7f1ac00d11e0 2024-03-29T11:17:27.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.205+0000 7f1ac4ff9640 1 -- 172.21.15.18:0/675592271 <== mon.1 v1:172.21.15.70:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f1ab4003750 con 0x7f1ac00d11e0 2024-03-29T11:17:27.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.206+0000 7f1ac4ff9640 1 -- 172.21.15.18:0/675592271 <== mon.1 v1:172.21.15.70:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f1ab40835c0 con 0x7f1ac00d11e0 2024-03-29T11:17:27.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.206+0000 7f1ac4ff9640 10 client.4465.objecter ms_dispatch 0x7f1ac8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:27.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.206+0000 7f1ac4ff9640 3 client.4465.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:27.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.206+0000 7f1ac4ff9640 3 client.4465.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:27.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.206+0000 7f1ac4ff9640 20 client.4465.objecter dump_active .. 0 homeless 2024-03-29T11:17:27.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.207+0000 7f1ac7fff640 10 monclient: get_auth_request con 0x7f1aac06e0b0 auth_method 0 2024-03-29T11:17:27.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.207+0000 7f1ac4ff9640 10 client.4465.objecter ms_handle_connect 0x7f1aac06e0b0 2024-03-29T11:17:27.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.211+0000 7f1ac4ff9640 1 -- 172.21.15.18:0/675592271 <== mon.1 v1:172.21.15.70:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1ab404ecd0 con 0x7f1ac00d11e0 2024-03-29T11:17:27.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.211+0000 7f1ac4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:27.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.211+0000 7f1ac4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:27.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.560+0000 7f1acd077640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:27.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.560+0000 7f1acd077640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:27.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.560+0000 7f1acd077640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:17:27.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.560+0000 7f1acd077640 1 -- 172.21.15.18:0/675592271 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ac0004cd0 con 0x7f1ac00d11e0 2024-03-29T11:17:27.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.562+0000 7f1ac4ff9640 1 -- 172.21.15.18:0/675592271 <== mon.1 v1:172.21.15.70:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f1ab4050dd0 con 0x7f1ac00d11e0 2024-03-29T11:17:27.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.562+0000 7f1ac4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:27.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.562+0000 7f1ac4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:27.540 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:17:27.541 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:27.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.566+0000 7f1acd077640 10 librados: watch_flush enter 2024-03-29T11:17:27.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.566+0000 7f1acd077640 10 librados: watch_flush exit 2024-03-29T11:17:27.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.566+0000 7f1acd077640 20 client.4465.objecter shutdown clearing up homeless session... 2024-03-29T11:17:27.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.566+0000 7f1acd077640 10 client.4465.objecter successfully canceled tick 2024-03-29T11:17:27.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.566+0000 7f1acd077640 1 -- 172.21.15.18:0/675592271 >> v1:172.21.15.70:6816/3101871562 conn(0x7f1aac06e0b0 legacy=0x7f1aac070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:27.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.566+0000 7f1acd077640 10 monclient: shutdown 2024-03-29T11:17:27.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.566+0000 7f1acd077640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:27.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.566+0000 7f1acd077640 1 -- 172.21.15.18:0/675592271 >> v1:172.21.15.70:6789/0 conn(0x7f1ac00d11e0 legacy=0x7f1ac016b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:27.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.566+0000 7f1acd077640 1 -- 172.21.15.18:0/675592271 shutdown_connections 2024-03-29T11:17:27.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.566+0000 7f1acd077640 1 -- 172.21.15.18:0/675592271 >> 172.21.15.18:0/675592271 conn(0x7f1ac0005c20 msgr2=0x7f1ac00c1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:27.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.567+0000 7f1acd077640 1 -- 172.21.15.18:0/675592271 shutdown_connections 2024-03-29T11:17:27.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.567+0000 7f1acd077640 1 -- 172.21.15.18:0/675592271 wait complete. 2024-03-29T11:17:27.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:27.567+0000 7f1acd077640 1 librados: shutdown 2024-03-29T11:17:27.563 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:34.371 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:34.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.556+0000 7feda0eef640 10 monclient: get_monmap_and_config 2024-03-29T11:17:34.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.556+0000 7feda0eef640 10 monclient: build_initial_monmap 2024-03-29T11:17:34.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.557+0000 7feda0eef640 10 monclient: monmap: 2024-03-29T11:17:34.534 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:17:34.534 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:34.534 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:17:34.557908+0000 2024-03-29T11:17:34.535 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:17:34.557908+0000 2024-03-29T11:17:34.535 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:34.535 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:34.535 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:17:34.535 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:17:34.535 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:17:34.535 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:34.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.557+0000 7feda0eef640 1 Processor -- start 2024-03-29T11:17:34.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.557+0000 7feda0eef640 1 -- start start 2024-03-29T11:17:34.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.557+0000 7feda0eef640 10 monclient: init 2024-03-29T11:17:34.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.559+0000 7feda0eef640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:34.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.559+0000 7feda0eef640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:17:34.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.559+0000 7feda0eef640 10 monclient(hunting): picked mon.noname-a con 0x7fed9c0b0770 addr v1:172.21.15.18:6789/0 2024-03-29T11:17:34.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.559+0000 7feda0eef640 10 monclient(hunting): picked mon.noname-c con 0x7fed9c14a700 addr v1:172.21.15.70:6789/0 2024-03-29T11:17:34.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.559+0000 7feda0eef640 10 monclient(hunting): picked mon.noname-b con 0x7fed9c0ac5a0 addr v1:172.21.15.18:6790/0 2024-03-29T11:17:34.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.559+0000 7feda0eef640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed9c0aba70 con 0x7fed9c0b0770 2024-03-29T11:17:34.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.559+0000 7feda0eef640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed9c0aecb0 con 0x7fed9c14a700 2024-03-29T11:17:34.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.559+0000 7feda0eef640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed9c0af150 con 0x7fed9c0ac5a0 2024-03-29T11:17:34.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.559+0000 7feda0eef640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:34.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.559+0000 7feda0eef640 10 monclient(hunting): authenticate will time out at 1280.831456s 2024-03-29T11:17:34.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.559+0000 7fed9a59c640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fed9c0b0770 0x7fed9c149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:52500/0 (socket says 172.21.15.18:52500) 2024-03-29T11:17:34.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.559+0000 7fed9a59c640 1 -- 172.21.15.18:0/1686680036 learned_addr learned my addr 172.21.15.18:0/1686680036 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:17:34.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.560+0000 7fed9959a640 1 -- 172.21.15.18:0/1686680036 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2006786373 0 0) 0x7fed9c0aba70 con 0x7fed9c0b0770 2024-03-29T11:17:34.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.560+0000 7fed9959a640 10 monclient(hunting): _negotiate 2024-03-29T11:17:34.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.560+0000 7fed9959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:34.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.560+0000 7fed9959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:34.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.560+0000 7fed9959a640 10 monclient(hunting): my global_id is 4500 2024-03-29T11:17:34.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.560+0000 7fed9959a640 1 -- 172.21.15.18:0/1686680036 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed88003520 con 0x7fed9c0b0770 2024-03-29T11:17:34.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.560+0000 7fed9959a640 1 -- 172.21.15.18:0/1686680036 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 261403045 0 0) 0x7fed9c0af150 con 0x7fed9c0ac5a0 2024-03-29T11:17:34.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.560+0000 7fed9959a640 10 monclient(hunting): _negotiate 2024-03-29T11:17:34.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.560+0000 7fed9959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:34.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.560+0000 7fed9959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:34.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.560+0000 7fed9959a640 10 monclient(hunting): my global_id is 4586 2024-03-29T11:17:34.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.560+0000 7fed9959a640 1 -- 172.21.15.18:0/1686680036 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed9c0aba70 con 0x7fed9c0ac5a0 2024-03-29T11:17:34.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.560+0000 7fed9959a640 1 -- 172.21.15.18:0/1686680036 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 369874191 0 0) 0x7fed9c0aecb0 con 0x7fed9c14a700 2024-03-29T11:17:34.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.560+0000 7fed9959a640 10 monclient(hunting): _negotiate 2024-03-29T11:17:34.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.560+0000 7fed9959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:34.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.560+0000 7fed9959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:34.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.560+0000 7fed9959a640 10 monclient(hunting): my global_id is 4468 2024-03-29T11:17:34.541 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.560+0000 7fed9959a640 1 -- 172.21.15.18:0/1686680036 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed9c0af150 con 0x7fed9c14a700 2024-03-29T11:17:34.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.560+0000 7fed9959a640 1 -- 172.21.15.18:0/1686680036 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1521836886 0 0) 0x7fed9c0aba70 con 0x7fed9c0ac5a0 2024-03-29T11:17:34.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.560+0000 7fed9959a640 1 -- 172.21.15.18:0/1686680036 >> v1:172.21.15.70:6789/0 conn(0x7fed9c14a700 legacy=0x7fed9c0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:34.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.560+0000 7fed9959a640 1 -- 172.21.15.18:0/1686680036 >> v1:172.21.15.18:6789/0 conn(0x7fed9c0b0770 legacy=0x7fed9c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:34.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.560+0000 7fed9959a640 10 monclient: _finish_hunting 0 2024-03-29T11:17:34.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.560+0000 7fed9959a640 1 monclient: found mon.noname-b 2024-03-29T11:17:34.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.560+0000 7fed9959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:34.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.560+0000 7fed9959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:17:34.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.560+0000 7fed9959a640 1 -- 172.21.15.18:0/1686680036 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed9c0ac9d0 con 0x7fed9c0ac5a0 2024-03-29T11:17:34.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.561+0000 7fed9959a640 10 monclient: _finish_auth 0 2024-03-29T11:17:34.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.561+0000 7fed9959a640 10 monclient: _check_auth_tickets 2024-03-29T11:17:34.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.561+0000 7fed9959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:34.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.561+0000 7fed9959a640 1 -- 172.21.15.18:0/1686680036 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fed940019f0 con 0x7fed9c0ac5a0 2024-03-29T11:17:34.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.561+0000 7fed9959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:34.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.561+0000 7fed9959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:17:34.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.561+0000 7fed9959a640 10 monclient: dump: 2024-03-29T11:17:34.543 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:17:34.543 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:34.543 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:34.543 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:34.543 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:34.543 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:34.543 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:17:34.543 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:17:34.543 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:17:34.543 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:34.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.561+0000 7fed9959a640 1 -- 172.21.15.18:0/1686680036 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed94001c40 con 0x7fed9c0ac5a0 2024-03-29T11:17:34.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.561+0000 7fed9959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:34.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.561+0000 7feda0eef640 5 monclient: authenticate success, global_id 4586 2024-03-29T11:17:34.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.561+0000 7feda0eef640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:34.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.561+0000 7feda0eef640 4 set_mon_vals no callback set 2024-03-29T11:17:34.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.561+0000 7fed9959a640 1 -- 172.21.15.18:0/1686680036 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fed94001fd0 con 0x7fed9c0ac5a0 2024-03-29T11:17:34.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.561+0000 7fed9959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:34.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.561+0000 7fed9959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:34.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.561+0000 7fed9959a640 10 monclient: dump: 2024-03-29T11:17:34.544 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:17:34.544 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:34.544 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:34.544 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:34.544 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:34.544 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:34.544 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:17:34.544 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:17:34.544 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:17:34.544 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:34.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.561+0000 7feda0eef640 10 monclient: shutdown 2024-03-29T11:17:34.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.561+0000 7feda0eef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:34.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.561+0000 7feda0eef640 1 -- 172.21.15.18:0/1686680036 >> v1:172.21.15.18:6790/0 conn(0x7fed9c0ac5a0 legacy=0x7fed9c151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:34.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.562+0000 7feda0eef640 1 -- 172.21.15.18:0/1686680036 shutdown_connections 2024-03-29T11:17:34.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.562+0000 7feda0eef640 1 -- 172.21.15.18:0/1686680036 >> 172.21.15.18:0/1686680036 conn(0x7fed9c06b660 msgr2=0x7fed9c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:34.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.562+0000 7feda0eef640 1 -- 172.21.15.18:0/1686680036 shutdown_connections 2024-03-29T11:17:34.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.562+0000 7feda0eef640 1 -- 172.21.15.18:0/1686680036 wait complete. 2024-03-29T11:17:34.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.563+0000 7feda0eef640 10 monclient: build_initial_monmap 2024-03-29T11:17:34.548 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.563+0000 7feda0eef640 10 monclient: monmap: 2024-03-29T11:17:34.548 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:17:34.549 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:34.549 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:17:34.549 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:17:34.549 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:34.549 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:34.549 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:17:34.549 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:17:34.549 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:17:34.549 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:34.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.564+0000 7feda0eef640 1 librados: starting msgr at 2024-03-29T11:17:34.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.564+0000 7feda0eef640 1 librados: starting objecter 2024-03-29T11:17:34.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.564+0000 7feda0eef640 1 Processor -- start 2024-03-29T11:17:34.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.564+0000 7feda0eef640 1 -- start start 2024-03-29T11:17:34.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.564+0000 7feda0eef640 1 librados: setting wanted keys 2024-03-29T11:17:34.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.564+0000 7feda0eef640 1 librados: calling monclient init 2024-03-29T11:17:34.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.564+0000 7feda0eef640 10 monclient: init 2024-03-29T11:17:34.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.565+0000 7feda0eef640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:34.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.565+0000 7feda0eef640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:17:34.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.566+0000 7fed9a59c640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fed9c0ac5a0 0x7fed9c11e1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:51724/0 (socket says 172.21.15.18:51724) 2024-03-29T11:17:34.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.566+0000 7fed9a59c640 1 -- 172.21.15.18:0/3238573519 learned_addr learned my addr 172.21.15.18:0/3238573519 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:17:34.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.566+0000 7feda0eef640 10 monclient(hunting): picked mon.noname-c con 0x7fed9c0ac5a0 addr v1:172.21.15.18:6790/0 2024-03-29T11:17:34.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.566+0000 7feda0eef640 10 monclient(hunting): picked mon.noname-a con 0x7fed9c0b0770 addr v1:172.21.15.18:6789/0 2024-03-29T11:17:34.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.566+0000 7feda0eef640 10 monclient(hunting): picked mon.noname-b con 0x7fed9c14a700 addr v1:172.21.15.70:6789/0 2024-03-29T11:17:34.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.566+0000 7feda0eef640 1 -- 172.21.15.18:0/3238573519 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed9c11fc30 con 0x7fed9c0b0770 2024-03-29T11:17:34.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.567+0000 7feda0eef640 1 -- 172.21.15.18:0/3238573519 --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed9c11b1c0 con 0x7fed9c14a700 2024-03-29T11:17:34.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.566+0000 7fed92ffd640 10 client.?.objecter ms_handle_connect 0x7fed9c0b0770 2024-03-29T11:17:34.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.567+0000 7fed92ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:34.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.567+0000 7feda0eef640 1 -- 172.21.15.18:0/3238573519 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed9c11b350 con 0x7fed9c0ac5a0 2024-03-29T11:17:34.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.567+0000 7feda0eef640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:34.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.567+0000 7feda0eef640 10 monclient(hunting): authenticate will time out at 1280.839050s 2024-03-29T11:17:34.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.567+0000 7fed92ffd640 10 client.?.objecter ms_handle_connect 0x7fed9c0ac5a0 2024-03-29T11:17:34.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.567+0000 7fed92ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:34.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.567+0000 7fed92ffd640 10 client.?.objecter ms_handle_connect 0x7fed9c14a700 2024-03-29T11:17:34.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.567+0000 7fed92ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:34.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.567+0000 7fed92ffd640 1 -- 172.21.15.18:0/3238573519 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3713844171 0 0) 0x7fed9c11fc30 con 0x7fed9c0b0770 2024-03-29T11:17:34.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.567+0000 7fed92ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:17:34.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.567+0000 7fed92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:34.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.567+0000 7fed92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:34.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.567+0000 7fed92ffd640 10 monclient(hunting): my global_id is 4503 2024-03-29T11:17:34.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.567+0000 7fed92ffd640 1 -- 172.21.15.18:0/3238573519 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed7c004eb0 con 0x7fed9c0b0770 2024-03-29T11:17:34.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.567+0000 7fed92ffd640 1 -- 172.21.15.18:0/3238573519 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4292540911 0 0) 0x7fed9c11b1c0 con 0x7fed9c14a700 2024-03-29T11:17:34.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.567+0000 7fed92ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:17:34.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.567+0000 7fed92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:34.551 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.567+0000 7fed92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:34.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.567+0000 7fed92ffd640 10 monclient(hunting): my global_id is 4471 2024-03-29T11:17:34.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.567+0000 7fed92ffd640 1 -- 172.21.15.18:0/3238573519 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed9c11fc30 con 0x7fed9c14a700 2024-03-29T11:17:34.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.567+0000 7fed92ffd640 1 -- 172.21.15.18:0/3238573519 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3078019013 0 0) 0x7fed88003520 con 0x7fed9c0ac5a0 2024-03-29T11:17:34.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.567+0000 7fed92ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:17:34.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.567+0000 7fed92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:34.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.567+0000 7fed92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:34.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.567+0000 7fed92ffd640 10 monclient(hunting): my global_id is 4589 2024-03-29T11:17:34.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.567+0000 7fed92ffd640 1 -- 172.21.15.18:0/3238573519 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed9c11b1c0 con 0x7fed9c0ac5a0 2024-03-29T11:17:34.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.567+0000 7fed92ffd640 1 -- 172.21.15.18:0/3238573519 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1973827502 0 0) 0x7fed7c004eb0 con 0x7fed9c0b0770 2024-03-29T11:17:34.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.567+0000 7fed92ffd640 1 -- 172.21.15.18:0/3238573519 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed88003520 con 0x7fed9c0b0770 2024-03-29T11:17:34.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.567+0000 7fed92ffd640 1 -- 172.21.15.18:0/3238573519 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fed8c001890 con 0x7fed9c0b0770 2024-03-29T11:17:34.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.567+0000 7fed92ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:34.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.567+0000 7fed92ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:17:34.552 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.567+0000 7fed92ffd640 10 monclient(hunting): dump: 2024-03-29T11:17:34.553 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:17:34.553 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:34.553 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:34.553 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:34.553 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:34.553 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:34.553 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:17:34.553 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:17:34.553 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:17:34.553 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:34.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.568+0000 7fed92ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:34.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.568+0000 7fed92ffd640 1 -- 172.21.15.18:0/3238573519 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3082747966 0 0) 0x7fed9c11fc30 con 0x7fed9c14a700 2024-03-29T11:17:34.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.568+0000 7fed92ffd640 1 -- 172.21.15.18:0/3238573519 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed7c004eb0 con 0x7fed9c14a700 2024-03-29T11:17:34.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.568+0000 7fed92ffd640 1 -- 172.21.15.18:0/3238573519 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fed94001810 con 0x7fed9c14a700 2024-03-29T11:17:34.553 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.568+0000 7fed92ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:34.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.568+0000 7fed92ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:17:34.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.568+0000 7fed92ffd640 10 monclient(hunting): dump: 2024-03-29T11:17:34.554 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:17:34.554 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:34.554 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:34.554 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:34.554 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:34.554 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:34.554 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:17:34.554 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:17:34.554 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:17:34.554 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:34.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.568+0000 7fed92ffd640 1 -- 172.21.15.18:0/3238573519 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2510055864 0 0) 0x7fed88003520 con 0x7fed9c0b0770 2024-03-29T11:17:34.554 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.568+0000 7fed92ffd640 1 -- 172.21.15.18:0/3238573519 >> v1:172.21.15.18:6790/0 conn(0x7fed9c0ac5a0 legacy=0x7fed9c11e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:34.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.568+0000 7fed92ffd640 1 -- 172.21.15.18:0/3238573519 >> v1:172.21.15.70:6789/0 conn(0x7fed9c14a700 legacy=0x7fed9c1e7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:34.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.568+0000 7fed92ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:17:34.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.568+0000 7fed92ffd640 1 monclient: found mon.a 2024-03-29T11:17:34.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.568+0000 7fed92ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:34.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.568+0000 7fed92ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:17:34.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.568+0000 7fed92ffd640 1 -- 172.21.15.18:0/3238573519 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed9c11e8b0 con 0x7fed9c0b0770 2024-03-29T11:17:34.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.568+0000 7fed92ffd640 10 monclient: _finish_auth 0 2024-03-29T11:17:34.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.568+0000 7fed92ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:17:34.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.568+0000 7fed92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:34.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.568+0000 7feda0eef640 5 monclient: authenticate success, global_id 4503 2024-03-29T11:17:34.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.568+0000 7feda0eef640 10 monclient: _renew_subs 2024-03-29T11:17:34.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.568+0000 7feda0eef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:17:34.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.568+0000 7feda0eef640 1 -- 172.21.15.18:0/3238573519 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed9c11eb20 con 0x7fed9c0b0770 2024-03-29T11:17:34.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.568+0000 7feda0eef640 10 client.4503.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:34.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.568+0000 7feda0eef640 10 monclient: _renew_subs 2024-03-29T11:17:34.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.568+0000 7feda0eef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:17:34.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.568+0000 7feda0eef640 1 -- 172.21.15.18:0/3238573519 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fed9c1e17b0 con 0x7fed9c0b0770 2024-03-29T11:17:34.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.568+0000 7feda0eef640 1 librados: init done 2024-03-29T11:17:34.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.569+0000 7fed92ffd640 1 -- 172.21.15.18:0/3238573519 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed8c0035f0 con 0x7fed9c0b0770 2024-03-29T11:17:34.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.569+0000 7fed92ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:34.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.569+0000 7fed92ffd640 1 -- 172.21.15.18:0/3238573519 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fed8c003980 con 0x7fed9c0b0770 2024-03-29T11:17:34.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.569+0000 7fed92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:34.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.569+0000 7fed92ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:17:34.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.569+0000 7fed92ffd640 10 monclient: dump: 2024-03-29T11:17:34.556 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:17:34.556 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:34.556 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:34.556 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:34.556 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:34.556 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:34.556 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:17:34.557 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:17:34.557 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:17:34.557 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:34.557 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.569+0000 7fed93fff640 4 set_mon_vals no callback set 2024-03-29T11:17:34.557 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.571+0000 7feda0eef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:34.557 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.571+0000 7feda0eef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:34.557 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.571+0000 7feda0eef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:17:34.557 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.571+0000 7feda0eef640 1 -- 172.21.15.18:0/3238573519 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed80005080 con 0x7fed9c0b0770 2024-03-29T11:17:34.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.571+0000 7fed92ffd640 1 -- 172.21.15.18:0/3238573519 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7fed8c003ae0 con 0x7fed9c0b0770 2024-03-29T11:17:34.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.572+0000 7fed92ffd640 1 -- 172.21.15.18:0/3238573519 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7fed8c0837a0 con 0x7fed9c0b0770 2024-03-29T11:17:34.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.572+0000 7fed92ffd640 10 client.4503.objecter ms_dispatch 0x7fed9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:34.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.572+0000 7fed92ffd640 3 client.4503.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:34.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.572+0000 7fed92ffd640 3 client.4503.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:34.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.572+0000 7fed92ffd640 20 client.4503.objecter dump_active .. 0 homeless 2024-03-29T11:17:34.558 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.572+0000 7fed9a59c640 10 monclient: get_auth_request con 0x7fed7c06dd80 auth_method 0 2024-03-29T11:17:34.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.572+0000 7fed92ffd640 10 client.4503.objecter ms_handle_connect 0x7fed7c06dd80 2024-03-29T11:17:34.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.579+0000 7fed92ffd640 1 -- 172.21.15.18:0/3238573519 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fed8c04f080 con 0x7fed9c0b0770 2024-03-29T11:17:34.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.579+0000 7fed92ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:34.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.579+0000 7fed92ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:34.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.948+0000 7feda0eef640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:34.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.948+0000 7feda0eef640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:34.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.948+0000 7feda0eef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:17:34.926 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.948+0000 7feda0eef640 1 -- 172.21.15.18:0/3238573519 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed80005330 con 0x7fed9c0b0770 2024-03-29T11:17:34.926 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.949+0000 7fed92ffd640 1 -- 172.21.15.18:0/3238573519 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7fed8c050fb0 con 0x7fed9c0b0770 2024-03-29T11:17:34.926 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.949+0000 7fed92ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:34.926 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.949+0000 7fed92ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:34.927 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:17:34.927 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:34.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.952+0000 7feda0eef640 10 librados: watch_flush enter 2024-03-29T11:17:34.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.952+0000 7feda0eef640 10 librados: watch_flush exit 2024-03-29T11:17:34.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.952+0000 7feda0eef640 20 client.4503.objecter shutdown clearing up homeless session... 2024-03-29T11:17:34.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.952+0000 7feda0eef640 10 client.4503.objecter successfully canceled tick 2024-03-29T11:17:34.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.952+0000 7feda0eef640 1 -- 172.21.15.18:0/3238573519 >> v1:172.21.15.70:6816/3101871562 conn(0x7fed7c06dd80 legacy=0x7fed7c070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:34.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.952+0000 7feda0eef640 10 monclient: shutdown 2024-03-29T11:17:34.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.952+0000 7feda0eef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:34.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.952+0000 7feda0eef640 1 -- 172.21.15.18:0/3238573519 >> v1:172.21.15.18:6789/0 conn(0x7fed9c0b0770 legacy=0x7fed9c1e0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:34.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.952+0000 7feda0eef640 1 -- 172.21.15.18:0/3238573519 shutdown_connections 2024-03-29T11:17:34.930 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.952+0000 7feda0eef640 1 -- 172.21.15.18:0/3238573519 >> 172.21.15.18:0/3238573519 conn(0x7fed9c06b660 msgr2=0x7fed9c149480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:34.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.952+0000 7feda0eef640 1 -- 172.21.15.18:0/3238573519 shutdown_connections 2024-03-29T11:17:34.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.953+0000 7feda0eef640 1 -- 172.21.15.18:0/3238573519 wait complete. 2024-03-29T11:17:34.931 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:34.953+0000 7feda0eef640 1 librados: shutdown 2024-03-29T11:17:34.945 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:41.752 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:41.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.922+0000 7f8f06e86640 10 monclient: get_monmap_and_config 2024-03-29T11:17:41.899 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.922+0000 7f8f06e86640 10 monclient: build_initial_monmap 2024-03-29T11:17:41.900 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.922+0000 7f8f06e86640 10 monclient: monmap: 2024-03-29T11:17:41.900 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:17:41.900 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:41.900 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:17:41.923424+0000 2024-03-29T11:17:41.900 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:17:41.923424+0000 2024-03-29T11:17:41.900 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:41.900 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:41.900 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:17:41.900 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:17:41.900 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:17:41.900 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:41.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.923+0000 7f8f06e86640 1 Processor -- start 2024-03-29T11:17:41.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.924+0000 7f8f06e86640 1 -- start start 2024-03-29T11:17:41.901 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.924+0000 7f8f06e86640 10 monclient: init 2024-03-29T11:17:41.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.925+0000 7f8f06e86640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:41.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.925+0000 7f8f06e86640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:17:41.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.925+0000 7f8f06e86640 10 monclient(hunting): picked mon.noname-a con 0x7f8f000b0770 addr v1:172.21.15.18:6789/0 2024-03-29T11:17:41.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.925+0000 7f8f06e86640 10 monclient(hunting): picked mon.noname-b con 0x7f8f0014a700 addr v1:172.21.15.18:6790/0 2024-03-29T11:17:41.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.925+0000 7f8f06e86640 10 monclient(hunting): picked mon.noname-c con 0x7f8f000ac5a0 addr v1:172.21.15.70:6789/0 2024-03-29T11:17:41.902 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.925+0000 7f8f06e86640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f000aba70 con 0x7f8f000b0770 2024-03-29T11:17:41.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.925+0000 7f8f06e86640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f000aecb0 con 0x7f8f000ac5a0 2024-03-29T11:17:41.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.925+0000 7f8f06e86640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f000af150 con 0x7f8f0014a700 2024-03-29T11:17:41.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.925+0000 7f8f06e86640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:41.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.925+0000 7f8f06e86640 10 monclient(hunting): authenticate will time out at 1288.197446s 2024-03-29T11:17:41.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.926+0000 7f8f05423640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f8f000ac5a0 0x7f8f00151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.18:38372/0 (socket says 172.21.15.18:38372) 2024-03-29T11:17:41.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.926+0000 7f8f05423640 1 -- 172.21.15.18:0/1239717930 learned_addr learned my addr 172.21.15.18:0/1239717930 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:17:41.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.926+0000 7f8eff7fe640 1 -- 172.21.15.18:0/1239717930 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3448177540 0 0) 0x7f8f000aecb0 con 0x7f8f000ac5a0 2024-03-29T11:17:41.903 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.926+0000 7f8eff7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:17:41.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.926+0000 7f8eff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:41.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.926+0000 7f8eff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:41.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.926+0000 7f8eff7fe640 10 monclient(hunting): my global_id is 4474 2024-03-29T11:17:41.904 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.926+0000 7f8eff7fe640 1 -- 172.21.15.18:0/1239717930 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ee8003520 con 0x7f8f000ac5a0 2024-03-29T11:17:41.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.927+0000 7f8eff7fe640 1 -- 172.21.15.18:0/1239717930 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3320751753 0 0) 0x7f8f000af150 con 0x7f8f0014a700 2024-03-29T11:17:41.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.927+0000 7f8eff7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:17:41.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.927+0000 7f8eff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:41.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.927+0000 7f8eff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:41.905 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.927+0000 7f8eff7fe640 10 monclient(hunting): my global_id is 4592 2024-03-29T11:17:41.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.927+0000 7f8eff7fe640 1 -- 172.21.15.18:0/1239717930 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f000aecb0 con 0x7f8f0014a700 2024-03-29T11:17:41.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.927+0000 7f8eff7fe640 1 -- 172.21.15.18:0/1239717930 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3285613419 0 0) 0x7f8f000aba70 con 0x7f8f000b0770 2024-03-29T11:17:41.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.927+0000 7f8eff7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:17:41.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.927+0000 7f8eff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:41.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.927+0000 7f8eff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:41.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.927+0000 7f8eff7fe640 10 monclient(hunting): my global_id is 4506 2024-03-29T11:17:41.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.927+0000 7f8eff7fe640 1 -- 172.21.15.18:0/1239717930 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f000af150 con 0x7f8f000b0770 2024-03-29T11:17:41.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.927+0000 7f8eff7fe640 1 -- 172.21.15.18:0/1239717930 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2201857588 0 0) 0x7f8ee8003520 con 0x7f8f000ac5a0 2024-03-29T11:17:41.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.927+0000 7f8eff7fe640 1 -- 172.21.15.18:0/1239717930 >> v1:172.21.15.18:6790/0 conn(0x7f8f0014a700 legacy=0x7f8f000abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:41.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.927+0000 7f8eff7fe640 1 -- 172.21.15.18:0/1239717930 >> v1:172.21.15.18:6789/0 conn(0x7f8f000b0770 legacy=0x7f8f00149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:41.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.927+0000 7f8eff7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:17:41.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.927+0000 7f8eff7fe640 1 monclient: found mon.noname-c 2024-03-29T11:17:41.906 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.927+0000 7f8eff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:41.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.927+0000 7f8eff7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.70:6789/0 2024-03-29T11:17:41.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.927+0000 7f8eff7fe640 1 -- 172.21.15.18:0/1239717930 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f000ac9d0 con 0x7f8f000ac5a0 2024-03-29T11:17:41.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.927+0000 7f8eff7fe640 10 monclient: _finish_auth 0 2024-03-29T11:17:41.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.927+0000 7f8eff7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:17:41.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.927+0000 7f8eff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:41.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.927+0000 7f8f06e86640 5 monclient: authenticate success, global_id 4474 2024-03-29T11:17:41.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.927+0000 7f8f06e86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:41.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.927+0000 7f8eff7fe640 1 -- 172.21.15.18:0/1239717930 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f8ef40015e0 con 0x7f8f000ac5a0 2024-03-29T11:17:41.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.927+0000 7f8eff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:41.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.927+0000 7f8eff7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:41.907 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.927+0000 7f8eff7fe640 10 monclient: dump: 2024-03-29T11:17:41.907 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:17:41.907 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:41.907 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:41.907 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:41.908 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:41.908 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:41.908 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:17:41.908 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:17:41.908 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:17:41.908 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:41.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.927+0000 7f8f06e86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:41.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.928+0000 7f8eff7fe640 1 -- 172.21.15.18:0/1239717930 <== mon.1 v1:172.21.15.70:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ef4001740 con 0x7f8f000ac5a0 2024-03-29T11:17:41.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.928+0000 7f8eff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:41.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.928+0000 7f8f06e86640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:41.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.928+0000 7f8f06e86640 4 set_mon_vals no callback set 2024-03-29T11:17:41.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.928+0000 7f8eff7fe640 1 -- 172.21.15.18:0/1239717930 <== mon.1 v1:172.21.15.70:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f8ef4001b50 con 0x7f8f000ac5a0 2024-03-29T11:17:41.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.928+0000 7f8eff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:41.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.928+0000 7f8eff7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:17:41.908 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.928+0000 7f8eff7fe640 10 monclient: dump: 2024-03-29T11:17:41.909 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:17:41.909 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:41.909 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:41.909 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:41.909 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:41.909 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:41.909 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:17:41.909 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:17:41.909 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:17:41.909 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:41.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.929+0000 7f8f06e86640 10 monclient: shutdown 2024-03-29T11:17:41.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.929+0000 7f8f06e86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:41.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.929+0000 7f8f06e86640 1 -- 172.21.15.18:0/1239717930 >> v1:172.21.15.70:6789/0 conn(0x7f8f000ac5a0 legacy=0x7f8f00151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:41.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.929+0000 7f8f06e86640 1 -- 172.21.15.18:0/1239717930 shutdown_connections 2024-03-29T11:17:41.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.929+0000 7f8f06e86640 1 -- 172.21.15.18:0/1239717930 >> 172.21.15.18:0/1239717930 conn(0x7f8f0006b660 msgr2=0x7f8f0006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:41.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.929+0000 7f8f06e86640 1 -- 172.21.15.18:0/1239717930 shutdown_connections 2024-03-29T11:17:41.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.929+0000 7f8f06e86640 1 -- 172.21.15.18:0/1239717930 wait complete. 2024-03-29T11:17:41.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.930+0000 7f8f06e86640 10 monclient: build_initial_monmap 2024-03-29T11:17:41.917 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.930+0000 7f8f06e86640 10 monclient: monmap: 2024-03-29T11:17:41.917 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:17:41.917 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:41.917 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:17:41.918 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:17:41.918 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:41.918 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:41.918 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:17:41.918 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:17:41.918 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:17:41.918 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:41.918 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.930+0000 7f8f06e86640 1 librados: starting msgr at 2024-03-29T11:17:41.918 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.930+0000 7f8f06e86640 1 librados: starting objecter 2024-03-29T11:17:41.918 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.930+0000 7f8f06e86640 1 Processor -- start 2024-03-29T11:17:41.918 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.930+0000 7f8f06e86640 1 -- start start 2024-03-29T11:17:41.918 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.930+0000 7f8f06e86640 1 librados: setting wanted keys 2024-03-29T11:17:41.918 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.930+0000 7f8f06e86640 1 librados: calling monclient init 2024-03-29T11:17:41.918 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.930+0000 7f8f06e86640 10 monclient: init 2024-03-29T11:17:41.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.931+0000 7f8f06e86640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:41.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.931+0000 7f8f06e86640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:17:41.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.931+0000 7f8f06e86640 10 monclient(hunting): picked mon.noname-a con 0x7f8f000ac5a0 addr v1:172.21.15.18:6789/0 2024-03-29T11:17:41.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.931+0000 7f8f06e86640 10 monclient(hunting): picked mon.noname-c con 0x7f8f000b0770 addr v1:172.21.15.18:6790/0 2024-03-29T11:17:41.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.932+0000 7f8f06e86640 10 monclient(hunting): picked mon.noname-b con 0x7f8f0014a700 addr v1:172.21.15.70:6789/0 2024-03-29T11:17:41.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.932+0000 7f8f06e86640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f00128360 con 0x7f8f000ac5a0 2024-03-29T11:17:41.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.932+0000 7f8f06e86640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f001285d0 con 0x7f8f0014a700 2024-03-29T11:17:41.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.932+0000 7f8f06e86640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f00126330 con 0x7f8f000b0770 2024-03-29T11:17:41.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.932+0000 7f8f06e86640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:41.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.932+0000 7f8f06e86640 10 monclient(hunting): authenticate will time out at 1288.204063s 2024-03-29T11:17:41.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.932+0000 7f8efffff640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f8f000b0770 0x7f8f00126850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:55106/0 (socket says 172.21.15.18:55106) 2024-03-29T11:17:41.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.932+0000 7f8efffff640 1 -- 172.21.15.18:0/3568674767 learned_addr learned my addr 172.21.15.18:0/3568674767 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:17:41.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.932+0000 7f8efd7fa640 10 client.?.objecter ms_handle_connect 0x7f8f000b0770 2024-03-29T11:17:41.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.932+0000 7f8efd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:41.919 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.932+0000 7f8efd7fa640 10 client.?.objecter ms_handle_connect 0x7f8f000ac5a0 2024-03-29T11:17:41.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.932+0000 7f8efd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:41.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.932+0000 7f8efd7fa640 1 -- 172.21.15.18:0/3568674767 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3869075439 0 0) 0x7f8f00126330 con 0x7f8f000b0770 2024-03-29T11:17:41.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.932+0000 7f8efd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:17:41.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.932+0000 7f8efd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:41.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.932+0000 7f8efd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:41.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.932+0000 7f8efd7fa640 10 monclient(hunting): my global_id is 4595 2024-03-29T11:17:41.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.932+0000 7f8efd7fa640 1 -- 172.21.15.18:0/3568674767 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8eec004eb0 con 0x7f8f000b0770 2024-03-29T11:17:41.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.932+0000 7f8efd7fa640 1 -- 172.21.15.18:0/3568674767 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 674345964 0 0) 0x7f8f00128360 con 0x7f8f000ac5a0 2024-03-29T11:17:41.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.932+0000 7f8efd7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:17:41.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.932+0000 7f8efd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:41.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.932+0000 7f8efd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:41.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.932+0000 7f8efd7fa640 10 monclient(hunting): my global_id is 4509 2024-03-29T11:17:41.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.932+0000 7f8efd7fa640 1 -- 172.21.15.18:0/3568674767 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f00126330 con 0x7f8f000ac5a0 2024-03-29T11:17:41.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.932+0000 7f8efd7fa640 1 -- 172.21.15.18:0/3568674767 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4135615795 0 0) 0x7f8eec004eb0 con 0x7f8f000b0770 2024-03-29T11:17:41.920 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.932+0000 7f8efd7fa640 1 -- 172.21.15.18:0/3568674767 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f00128360 con 0x7f8f000b0770 2024-03-29T11:17:41.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.932+0000 7f8efd7fa640 1 -- 172.21.15.18:0/3568674767 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f8ef0001990 con 0x7f8f000b0770 2024-03-29T11:17:41.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.932+0000 7f8efd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:41.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.932+0000 7f8efd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:41.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.932+0000 7f8efd7fa640 10 monclient(hunting): dump: 2024-03-29T11:17:41.921 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:17:41.921 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:41.921 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:41.921 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:41.921 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:41.921 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:41.921 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:17:41.921 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:17:41.921 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:17:41.921 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:41.921 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.932+0000 7f8efd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:41.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.933+0000 7f8efd7fa640 10 client.?.objecter ms_handle_connect 0x7f8f0014a700 2024-03-29T11:17:41.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.933+0000 7f8efd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:41.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.933+0000 7f8efd7fa640 1 -- 172.21.15.18:0/3568674767 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1848939574 0 0) 0x7f8f00126330 con 0x7f8f000ac5a0 2024-03-29T11:17:41.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.933+0000 7f8efd7fa640 1 -- 172.21.15.18:0/3568674767 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8eec004eb0 con 0x7f8f000ac5a0 2024-03-29T11:17:41.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.933+0000 7f8efd7fa640 1 -- 172.21.15.18:0/3568674767 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 641142891 0 0) 0x7f8f00128360 con 0x7f8f000b0770 2024-03-29T11:17:41.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.933+0000 7f8efd7fa640 1 -- 172.21.15.18:0/3568674767 >> v1:172.21.15.70:6789/0 conn(0x7f8f0014a700 legacy=0x7f8f00126f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:41.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.933+0000 7f8efd7fa640 1 -- 172.21.15.18:0/3568674767 >> v1:172.21.15.18:6789/0 conn(0x7f8f000ac5a0 legacy=0x7f8f00144ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:41.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.933+0000 7f8efd7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:17:41.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.933+0000 7f8efd7fa640 1 monclient: found mon.c 2024-03-29T11:17:41.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.933+0000 7f8efd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:41.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.933+0000 7f8efd7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:17:41.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.933+0000 7f8efd7fa640 1 -- 172.21.15.18:0/3568674767 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f00125f70 con 0x7f8f000b0770 2024-03-29T11:17:41.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.933+0000 7f8efd7fa640 10 monclient: _finish_auth 0 2024-03-29T11:17:41.922 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.933+0000 7f8efd7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:17:41.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.933+0000 7f8efd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:41.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.933+0000 7f8f06e86640 5 monclient: authenticate success, global_id 4595 2024-03-29T11:17:41.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.933+0000 7f8f06e86640 10 monclient: _renew_subs 2024-03-29T11:17:41.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.933+0000 7f8f06e86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:17:41.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.933+0000 7f8f06e86640 1 -- 172.21.15.18:0/3568674767 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f001dfad0 con 0x7f8f000b0770 2024-03-29T11:17:41.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.933+0000 7f8f06e86640 10 client.4595.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:41.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.933+0000 7f8f06e86640 10 monclient: _renew_subs 2024-03-29T11:17:41.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.933+0000 7f8f06e86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:17:41.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.933+0000 7f8f06e86640 1 -- 172.21.15.18:0/3568674767 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f001df240 con 0x7f8f000b0770 2024-03-29T11:17:41.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.934+0000 7f8f06e86640 1 librados: init done 2024-03-29T11:17:41.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.934+0000 7f8efd7fa640 1 -- 172.21.15.18:0/3568674767 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ef0001af0 con 0x7f8f000b0770 2024-03-29T11:17:41.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.935+0000 7f8efd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:41.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.935+0000 7f8efe7fc640 4 set_mon_vals no callback set 2024-03-29T11:17:41.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.935+0000 7f8f06e86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:41.923 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.936+0000 7f8efd7fa640 1 -- 172.21.15.18:0/3568674767 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f8ef00038c0 con 0x7f8f000b0770 2024-03-29T11:17:41.924 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.936+0000 7f8efd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:41.924 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.936+0000 7f8efd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:41.924 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.936+0000 7f8efd7fa640 10 monclient: dump: 2024-03-29T11:17:41.924 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:17:41.924 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:41.924 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:41.924 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:41.924 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:41.924 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:41.924 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:17:41.924 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:17:41.924 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:17:41.924 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:41.924 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.936+0000 7f8f06e86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:41.924 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.936+0000 7f8f06e86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:17:41.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.936+0000 7f8f06e86640 1 -- 172.21.15.18:0/3568674767 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ee0005080 con 0x7f8f000b0770 2024-03-29T11:17:41.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.937+0000 7f8efd7fa640 1 -- 172.21.15.18:0/3568674767 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f8ef0003aa0 con 0x7f8f000b0770 2024-03-29T11:17:41.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.938+0000 7f8f04c22640 10 monclient: get_auth_request con 0x7f8eec06dda0 auth_method 0 2024-03-29T11:17:41.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.938+0000 7f8efd7fa640 1 -- 172.21.15.18:0/3568674767 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f8ef0084a60 con 0x7f8f000b0770 2024-03-29T11:17:41.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.938+0000 7f8efd7fa640 10 client.4595.objecter ms_dispatch 0x7f8f00000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:41.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.938+0000 7f8efd7fa640 3 client.4595.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:41.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.938+0000 7f8efd7fa640 3 client.4595.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:41.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.939+0000 7f8efd7fa640 20 client.4595.objecter dump_active .. 0 homeless 2024-03-29T11:17:41.925 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.939+0000 7f8efd7fa640 10 client.4595.objecter ms_handle_connect 0x7f8eec06dda0 2024-03-29T11:17:41.926 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.944+0000 7f8efd7fa640 1 -- 172.21.15.18:0/3568674767 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8ef0050340 con 0x7f8f000b0770 2024-03-29T11:17:41.926 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.944+0000 7f8efd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:41.926 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:41.944+0000 7f8efd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:42.299 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:42.321+0000 7f8f06e86640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:42.299 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:42.321+0000 7f8f06e86640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:42.299 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:42.321+0000 7f8f06e86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:17:42.299 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:42.321+0000 7f8f06e86640 1 -- 172.21.15.18:0/3568674767 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ee0005330 con 0x7f8f000b0770 2024-03-29T11:17:42.299 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:42.322+0000 7f8efd7fa640 1 -- 172.21.15.18:0/3568674767 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f8ef0052270 con 0x7f8f000b0770 2024-03-29T11:17:42.299 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:42.322+0000 7f8efd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:42.299 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:42.322+0000 7f8efd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:42.300 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:17:42.300 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:42.306 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:42.328+0000 7f8f06e86640 10 librados: watch_flush enter 2024-03-29T11:17:42.306 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:42.328+0000 7f8f06e86640 10 librados: watch_flush exit 2024-03-29T11:17:42.306 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:42.328+0000 7f8f06e86640 20 client.4595.objecter shutdown clearing up homeless session... 2024-03-29T11:17:42.306 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:42.328+0000 7f8f06e86640 10 client.4595.objecter successfully canceled tick 2024-03-29T11:17:42.306 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:42.328+0000 7f8f06e86640 1 -- 172.21.15.18:0/3568674767 >> v1:172.21.15.70:6816/3101871562 conn(0x7f8eec06dda0 legacy=0x7f8eec070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:42.306 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:42.328+0000 7f8f06e86640 10 monclient: shutdown 2024-03-29T11:17:42.306 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:42.328+0000 7f8f06e86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:42.306 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:42.328+0000 7f8f06e86640 1 -- 172.21.15.18:0/3568674767 >> v1:172.21.15.18:6790/0 conn(0x7f8f000b0770 legacy=0x7f8f00126850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:42.306 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:42.328+0000 7f8f06e86640 1 -- 172.21.15.18:0/3568674767 shutdown_connections 2024-03-29T11:17:42.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:42.328+0000 7f8f06e86640 1 -- 172.21.15.18:0/3568674767 >> 172.21.15.18:0/3568674767 conn(0x7f8f0006b660 msgr2=0x7f8f000b11d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:42.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:42.329+0000 7f8f06e86640 1 -- 172.21.15.18:0/3568674767 shutdown_connections 2024-03-29T11:17:42.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:42.329+0000 7f8f06e86640 1 -- 172.21.15.18:0/3568674767 wait complete. 2024-03-29T11:17:42.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:42.329+0000 7f8f06e86640 1 librados: shutdown 2024-03-29T11:17:42.323 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:49.130 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:49.278 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.301+0000 7f9cf350e640 10 monclient: get_monmap_and_config 2024-03-29T11:17:49.278 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.301+0000 7f9cf350e640 10 monclient: build_initial_monmap 2024-03-29T11:17:49.278 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.301+0000 7f9cf350e640 10 monclient: monmap: 2024-03-29T11:17:49.279 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:17:49.279 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:49.279 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:17:49.302180+0000 2024-03-29T11:17:49.279 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:17:49.302180+0000 2024-03-29T11:17:49.279 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:49.279 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:49.279 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:17:49.279 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:17:49.279 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:17:49.279 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:49.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.304+0000 7f9cf350e640 1 Processor -- start 2024-03-29T11:17:49.281 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.304+0000 7f9cf350e640 1 -- start start 2024-03-29T11:17:49.282 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.304+0000 7f9cf350e640 10 monclient: init 2024-03-29T11:17:49.283 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.305+0000 7f9cf350e640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:49.283 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.306+0000 7f9cf350e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:17:49.286 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.306+0000 7f9cf350e640 10 monclient(hunting): picked mon.noname-c con 0x7f9cec0b4890 addr v1:172.21.15.70:6789/0 2024-03-29T11:17:49.286 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.309+0000 7f9cf350e640 10 monclient(hunting): picked mon.noname-a con 0x7f9cec13c040 addr v1:172.21.15.18:6789/0 2024-03-29T11:17:49.286 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.309+0000 7f9cf0aa9640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f9cec13c040 0x7f9cec1420d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:59908/0 (socket says 172.21.15.18:59908) 2024-03-29T11:17:49.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.310+0000 7f9cf12aa640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f9cec0b4890 0x7f9cec13b930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.18:53695/0 (socket says 172.21.15.18:53695) 2024-03-29T11:17:49.287 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.310+0000 7f9cf350e640 10 monclient(hunting): picked mon.noname-b con 0x7f9cec1427e0 addr v1:172.21.15.18:6790/0 2024-03-29T11:17:49.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.310+0000 7f9cf0aa9640 1 -- 172.21.15.18:0/2969135493 learned_addr learned my addr 172.21.15.18:0/2969135493 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:17:49.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.310+0000 7f9cf350e640 1 -- 172.21.15.18:0/2969135493 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cec0b42f0 con 0x7f9cec13c040 2024-03-29T11:17:49.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.310+0000 7f9cf350e640 1 -- 172.21.15.18:0/2969135493 --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cec0aa7a0 con 0x7f9cec0b4890 2024-03-29T11:17:49.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.310+0000 7f9cf350e640 1 -- 172.21.15.18:0/2969135493 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cec0aaa10 con 0x7f9cec1427e0 2024-03-29T11:17:49.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.310+0000 7f9cf350e640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:49.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.310+0000 7f9cf350e640 10 monclient(hunting): authenticate will time out at 1295.582601s 2024-03-29T11:17:49.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.311+0000 7f9ce3fff640 1 -- 172.21.15.18:0/2969135493 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4222739499 0 0) 0x7f9cec0b42f0 con 0x7f9cec13c040 2024-03-29T11:17:49.288 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.311+0000 7f9ce3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:17:49.289 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.311+0000 7f9ce3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:49.289 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.311+0000 7f9ce3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:49.289 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.311+0000 7f9ce3fff640 10 monclient(hunting): my global_id is 4512 2024-03-29T11:17:49.289 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.311+0000 7f9ce3fff640 1 -- 172.21.15.18:0/2969135493 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cd8003520 con 0x7f9cec13c040 2024-03-29T11:17:49.289 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.311+0000 7f9ce3fff640 1 -- 172.21.15.18:0/2969135493 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4220240805 0 0) 0x7f9cd8003520 con 0x7f9cec13c040 2024-03-29T11:17:49.289 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.311+0000 7f9ce3fff640 1 -- 172.21.15.18:0/2969135493 >> v1:172.21.15.18:6790/0 conn(0x7f9cec1427e0 legacy=0x7f9cec14d880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:17:49.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.313+0000 7f9ce3fff640 1 -- 172.21.15.18:0/2969135493 >> v1:172.21.15.70:6789/0 conn(0x7f9cec0b4890 legacy=0x7f9cec13b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:17:49.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.313+0000 7f9ce3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:17:49.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.313+0000 7f9ce3fff640 1 monclient: found mon.noname-a 2024-03-29T11:17:49.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.313+0000 7f9ce3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:49.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.313+0000 7f9ce3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-29T11:17:49.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.313+0000 7f9ce3fff640 1 -- 172.21.15.18:0/2969135493 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cec0aacf0 con 0x7f9cec13c040 2024-03-29T11:17:49.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.313+0000 7f9ce3fff640 10 monclient: _finish_auth 0 2024-03-29T11:17:49.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.313+0000 7f9ce3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:17:49.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.313+0000 7f9ce3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:49.293 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.313+0000 7f9ce3fff640 1 -- 172.21.15.18:0/2969135493 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f9cdc001690 con 0x7f9cec13c040 2024-03-29T11:17:49.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.313+0000 7f9cf350e640 5 monclient: authenticate success, global_id 4512 2024-03-29T11:17:49.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.313+0000 7f9cf350e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:49.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.314+0000 7f9ce3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:49.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.314+0000 7f9ce3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:17:49.294 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.314+0000 7f9ce3fff640 10 monclient: dump: 2024-03-29T11:17:49.294 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:17:49.294 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:49.294 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:49.294 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:49.294 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:49.294 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:49.294 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:17:49.294 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:17:49.295 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:17:49.295 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:49.295 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.314+0000 7f9ce3fff640 1 -- 172.21.15.18:0/2969135493 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9cdc0018b0 con 0x7f9cec13c040 2024-03-29T11:17:49.295 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.314+0000 7f9cf350e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:17:49.295 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.314+0000 7f9ce3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:49.295 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.314+0000 7f9ce3fff640 1 -- 172.21.15.18:0/2969135493 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f9cdc001c40 con 0x7f9cec13c040 2024-03-29T11:17:49.295 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.314+0000 7f9cf350e640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:49.295 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.314+0000 7f9cf350e640 4 set_mon_vals no callback set 2024-03-29T11:17:49.295 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.315+0000 7f9ce3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:49.295 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.315+0000 7f9ce3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:17:49.295 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.315+0000 7f9ce3fff640 10 monclient: dump: 2024-03-29T11:17:49.296 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:17:49.296 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:49.296 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:49.296 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:49.296 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:49.296 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:49.296 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:17:49.296 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:17:49.296 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:17:49.296 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:49.296 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.315+0000 7f9cf350e640 10 monclient: shutdown 2024-03-29T11:17:49.296 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.315+0000 7f9cf350e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:49.296 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.315+0000 7f9cf350e640 1 -- 172.21.15.18:0/2969135493 >> v1:172.21.15.18:6789/0 conn(0x7f9cec13c040 legacy=0x7f9cec1420d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:49.296 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.316+0000 7f9cf350e640 1 -- 172.21.15.18:0/2969135493 shutdown_connections 2024-03-29T11:17:49.296 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.316+0000 7f9cf350e640 1 -- 172.21.15.18:0/2969135493 >> 172.21.15.18:0/2969135493 conn(0x7f9cec06d180 msgr2=0x7f9cec066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:49.297 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.316+0000 7f9cf350e640 1 -- 172.21.15.18:0/2969135493 shutdown_connections 2024-03-29T11:17:49.297 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.316+0000 7f9cf350e640 1 -- 172.21.15.18:0/2969135493 wait complete. 2024-03-29T11:17:49.300 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.317+0000 7f9cf350e640 10 monclient: build_initial_monmap 2024-03-29T11:17:49.300 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.317+0000 7f9cf350e640 10 monclient: monmap: 2024-03-29T11:17:49.300 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:17:49.300 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:49.300 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:17:49.300 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:17:49.300 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:49.301 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:49.301 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:17:49.301 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:17:49.301 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:17:49.301 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:49.301 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.318+0000 7f9cf350e640 1 librados: starting msgr at 2024-03-29T11:17:49.301 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.318+0000 7f9cf350e640 1 librados: starting objecter 2024-03-29T11:17:49.302 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.319+0000 7f9cf350e640 1 Processor -- start 2024-03-29T11:17:49.302 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.319+0000 7f9cf350e640 1 -- start start 2024-03-29T11:17:49.302 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.319+0000 7f9cf350e640 1 librados: setting wanted keys 2024-03-29T11:17:49.302 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.319+0000 7f9cf350e640 1 librados: calling monclient init 2024-03-29T11:17:49.302 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.319+0000 7f9cf350e640 10 monclient: init 2024-03-29T11:17:49.302 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.320+0000 7f9cf350e640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:49.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.320+0000 7f9cf350e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:17:49.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.320+0000 7f9cf350e640 10 monclient(hunting): picked mon.noname-a con 0x7f9cec0b4890 addr v1:172.21.15.18:6789/0 2024-03-29T11:17:49.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.321+0000 7f9cf350e640 10 monclient(hunting): picked mon.noname-b con 0x7f9cec13c040 addr v1:172.21.15.70:6789/0 2024-03-29T11:17:49.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.321+0000 7f9cf12aa640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f9cec0b4890 0x7f9cec1d6c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:60116/0 (socket says 172.21.15.18:60116) 2024-03-29T11:17:49.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.321+0000 7f9cf12aa640 1 -- 172.21.15.18:0/4245489612 learned_addr learned my addr 172.21.15.18:0/4245489612 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:17:49.303 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.321+0000 7f9ce1ffb640 10 client.?.objecter ms_handle_connect 0x7f9cec0b4890 2024-03-29T11:17:49.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.321+0000 7f9ce1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:49.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.321+0000 7f9cf350e640 10 monclient(hunting): picked mon.noname-c con 0x7f9cec1427e0 addr v1:172.21.15.18:6790/0 2024-03-29T11:17:49.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.321+0000 7f9cf350e640 1 -- 172.21.15.18:0/4245489612 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cec0abd50 con 0x7f9cec0b4890 2024-03-29T11:17:49.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.321+0000 7f9cf350e640 1 -- 172.21.15.18:0/4245489612 --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cec0ac030 con 0x7f9cec13c040 2024-03-29T11:17:49.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.321+0000 7f9cf350e640 1 -- 172.21.15.18:0/4245489612 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cec0ac310 con 0x7f9cec1427e0 2024-03-29T11:17:49.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.321+0000 7f9ce1ffb640 1 -- 172.21.15.18:0/4245489612 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1537340682 0 0) 0x7f9cec0abd50 con 0x7f9cec0b4890 2024-03-29T11:17:49.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.321+0000 7f9cf350e640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:49.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.322+0000 7f9cf350e640 10 monclient(hunting): authenticate will time out at 1295.593967s 2024-03-29T11:17:49.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.322+0000 7f9ce1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:17:49.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.322+0000 7f9ce1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:49.304 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.322+0000 7f9ce1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:49.305 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.322+0000 7f9ce1ffb640 10 monclient(hunting): my global_id is 4515 2024-03-29T11:17:49.305 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.322+0000 7f9ce1ffb640 1 -- 172.21.15.18:0/4245489612 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cd0004eb0 con 0x7f9cec0b4890 2024-03-29T11:17:49.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.323+0000 7f9ce1ffb640 1 -- 172.21.15.18:0/4245489612 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2427674746 0 0) 0x7f9cd0004eb0 con 0x7f9cec0b4890 2024-03-29T11:17:49.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.323+0000 7f9ce1ffb640 1 -- 172.21.15.18:0/4245489612 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9cec0abd50 con 0x7f9cec0b4890 2024-03-29T11:17:49.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.323+0000 7f9ce1ffb640 1 -- 172.21.15.18:0/4245489612 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f9ce8001510 con 0x7f9cec0b4890 2024-03-29T11:17:49.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.323+0000 7f9ce1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:49.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.323+0000 7f9ce1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:17:49.307 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.324+0000 7f9ce1ffb640 10 monclient(hunting): dump: 2024-03-29T11:17:49.308 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:17:49.308 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:49.308 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:49.308 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:49.308 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:49.308 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:49.308 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:17:49.308 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:17:49.308 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:17:49.308 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:49.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.324+0000 7f9ce1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:49.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.324+0000 7f9ce1ffb640 1 -- 172.21.15.18:0/4245489612 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 524733243 0 0) 0x7f9cec0abd50 con 0x7f9cec0b4890 2024-03-29T11:17:49.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.324+0000 7f9ce1ffb640 1 -- 172.21.15.18:0/4245489612 >> v1:172.21.15.18:6790/0 conn(0x7f9cec1427e0 legacy=0x7f9cec1e2490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:17:49.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.327+0000 7f9ce1ffb640 1 -- 172.21.15.18:0/4245489612 >> v1:172.21.15.70:6789/0 conn(0x7f9cec13c040 legacy=0x7f9cec13ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:49.308 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.327+0000 7f9ce1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:17:49.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.327+0000 7f9ce1ffb640 1 monclient: found mon.a 2024-03-29T11:17:49.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.327+0000 7f9ce1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:49.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.327+0000 7f9ce1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:17:49.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.327+0000 7f9ce1ffb640 1 -- 172.21.15.18:0/4245489612 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cec1da8b0 con 0x7f9cec0b4890 2024-03-29T11:17:49.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.327+0000 7f9ce1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:17:49.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.327+0000 7f9ce1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:17:49.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.327+0000 7f9ce1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:49.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.327+0000 7f9ce1ffb640 10 client.?.objecter ms_handle_connect 0x7f9cec13c040 2024-03-29T11:17:49.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.327+0000 7f9ce1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:49.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.328+0000 7f9ce1ffb640 1 -- 172.21.15.18:0/4245489612 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ce80016e0 con 0x7f9cec0b4890 2024-03-29T11:17:49.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.328+0000 7f9ce1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:49.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.328+0000 7f9ce1ffb640 1 -- 172.21.15.18:0/4245489612 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f9ce8004ba0 con 0x7f9cec0b4890 2024-03-29T11:17:49.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.328+0000 7f9ce1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:49.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.328+0000 7f9ce1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:17:49.309 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.328+0000 7f9ce1ffb640 10 monclient: dump: 2024-03-29T11:17:49.309 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:17:49.309 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:49.310 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:49.310 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:49.310 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:49.310 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:49.310 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:17:49.310 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:17:49.310 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:17:49.310 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:49.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.328+0000 7f9cf350e640 5 monclient: authenticate success, global_id 4515 2024-03-29T11:17:49.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.328+0000 7f9cf350e640 10 monclient: _renew_subs 2024-03-29T11:17:49.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.328+0000 7f9cf350e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:17:49.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.328+0000 7f9ce2ffd640 4 set_mon_vals no callback set 2024-03-29T11:17:49.310 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.328+0000 7f9cf350e640 1 -- 172.21.15.18:0/4245489612 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cec1d7360 con 0x7f9cec0b4890 2024-03-29T11:17:49.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.329+0000 7f9cf350e640 10 client.4515.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:49.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.329+0000 7f9cf350e640 10 monclient: _renew_subs 2024-03-29T11:17:49.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.329+0000 7f9cf350e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:17:49.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.329+0000 7f9cf350e640 1 -- 172.21.15.18:0/4245489612 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cec1dafb0 con 0x7f9cec0b4890 2024-03-29T11:17:49.311 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.329+0000 7f9cf350e640 1 librados: init done 2024-03-29T11:17:49.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.331+0000 7f9ce1ffb640 1 -- 172.21.15.18:0/4245489612 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f9ce801ab70 con 0x7f9cec0b4890 2024-03-29T11:17:49.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.332+0000 7f9ce1ffb640 1 -- 172.21.15.18:0/4245489612 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f9ce8084ab0 con 0x7f9cec0b4890 2024-03-29T11:17:49.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.332+0000 7f9ce1ffb640 10 client.4515.objecter ms_dispatch 0x7f9cec000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:49.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.332+0000 7f9ce1ffb640 3 client.4515.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:49.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.332+0000 7f9ce1ffb640 3 client.4515.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:49.312 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.332+0000 7f9ce1ffb640 20 client.4515.objecter dump_active .. 0 homeless 2024-03-29T11:17:49.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.332+0000 7f9cf350e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:49.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.332+0000 7f9cf350e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:49.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.332+0000 7f9cf350e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:17:49.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.332+0000 7f9cf350e640 1 -- 172.21.15.18:0/4245489612 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cec1daa40 con 0x7f9cec0b4890 2024-03-29T11:17:49.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.333+0000 7f9cf0aa9640 10 monclient: get_auth_request con 0x7f9cd006dbd0 auth_method 0 2024-03-29T11:17:49.313 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.334+0000 7f9ce1ffb640 10 client.4515.objecter ms_handle_connect 0x7f9cd006dbd0 2024-03-29T11:17:49.319 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.342+0000 7f9ce1ffb640 1 -- 172.21.15.18:0/4245489612 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9ce80501c0 con 0x7f9cec0b4890 2024-03-29T11:17:49.319 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.342+0000 7f9ce1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:49.319 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.342+0000 7f9ce1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:49.657 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.678+0000 7f9cf350e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:49.657 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.678+0000 7f9cf350e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:49.657 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.678+0000 7f9cf350e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:17:49.657 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.678+0000 7f9cf350e640 1 -- 172.21.15.18:0/4245489612 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9cec1da6c0 con 0x7f9cec0b4890 2024-03-29T11:17:49.657 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.680+0000 7f9ce1ffb640 1 -- 172.21.15.18:0/4245489612 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f9ce80522c0 con 0x7f9cec0b4890 2024-03-29T11:17:49.658 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.680+0000 7f9ce1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:49.658 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.680+0000 7f9ce1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:49.660 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:17:49.660 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:49.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.685+0000 7f9cf350e640 10 librados: watch_flush enter 2024-03-29T11:17:49.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.685+0000 7f9cf350e640 10 librados: watch_flush exit 2024-03-29T11:17:49.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.685+0000 7f9cf350e640 20 client.4515.objecter shutdown clearing up homeless session... 2024-03-29T11:17:49.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.685+0000 7f9cf350e640 10 client.4515.objecter successfully canceled tick 2024-03-29T11:17:49.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.685+0000 7f9cf350e640 1 -- 172.21.15.18:0/4245489612 >> v1:172.21.15.70:6816/3101871562 conn(0x7f9cd006dbd0 legacy=0x7f9cd0070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:49.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.685+0000 7f9cf350e640 10 monclient: shutdown 2024-03-29T11:17:49.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.685+0000 7f9cf350e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:49.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.685+0000 7f9cf350e640 1 -- 172.21.15.18:0/4245489612 >> v1:172.21.15.18:6789/0 conn(0x7f9cec0b4890 legacy=0x7f9cec1d6c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:49.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.688+0000 7f9cf350e640 1 -- 172.21.15.18:0/4245489612 shutdown_connections 2024-03-29T11:17:49.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.688+0000 7f9cf350e640 1 -- 172.21.15.18:0/4245489612 >> 172.21.15.18:0/4245489612 conn(0x7f9cec06d180 msgr2=0x7f9cec0b6ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:49.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.688+0000 7f9cf350e640 1 -- 172.21.15.18:0/4245489612 shutdown_connections 2024-03-29T11:17:49.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.688+0000 7f9cf350e640 1 -- 172.21.15.18:0/4245489612 wait complete. 2024-03-29T11:17:49.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:49.688+0000 7f9cf350e640 1 librados: shutdown 2024-03-29T11:17:49.682 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:17:56.489 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:17:56.637 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.660+0000 7f279359e640 10 monclient: get_monmap_and_config 2024-03-29T11:17:56.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.660+0000 7f279359e640 10 monclient: build_initial_monmap 2024-03-29T11:17:56.638 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.660+0000 7f279359e640 10 monclient: monmap: 2024-03-29T11:17:56.638 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:17:56.638 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:56.638 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:17:56.661424+0000 2024-03-29T11:17:56.638 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:17:56.661424+0000 2024-03-29T11:17:56.638 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:56.638 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:56.638 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:17:56.638 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:17:56.638 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:17:56.638 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:56.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.661+0000 7f279359e640 1 Processor -- start 2024-03-29T11:17:56.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.661+0000 7f279359e640 1 -- start start 2024-03-29T11:17:56.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.661+0000 7f279359e640 10 monclient: init 2024-03-29T11:17:56.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.662+0000 7f279359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:56.639 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.662+0000 7f279359e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:17:56.644 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.665+0000 7f279359e640 10 monclient(hunting): picked mon.noname-c con 0x7f278c0389e0 addr v1:172.21.15.70:6789/0 2024-03-29T11:17:56.644 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.665+0000 7f279359e640 10 monclient(hunting): picked mon.noname-b con 0x7f278c038dc0 addr v1:172.21.15.18:6790/0 2024-03-29T11:17:56.644 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.665+0000 7f279359e640 10 monclient(hunting): picked mon.noname-a con 0x7f278c030660 addr v1:172.21.15.18:6789/0 2024-03-29T11:17:56.644 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.665+0000 7f279359e640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f278c0c54c0 con 0x7f278c030660 2024-03-29T11:17:56.644 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.665+0000 7f279359e640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f278c036430 con 0x7f278c0389e0 2024-03-29T11:17:56.644 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.665+0000 7f279359e640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f278c039840 con 0x7f278c038dc0 2024-03-29T11:17:56.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.665+0000 7f279359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:56.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.666+0000 7f279359e640 10 monclient(hunting): authenticate will time out at 1302.937924s 2024-03-29T11:17:56.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.666+0000 7f2792d9d640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f278c030660 0x7f278c0d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:45384/0 (socket says 172.21.15.18:45384) 2024-03-29T11:17:56.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.666+0000 7f2792d9d640 1 -- 172.21.15.18:0/2727981743 learned_addr learned my addr 172.21.15.18:0/2727981743 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:17:56.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.666+0000 7f279159a640 1 -- 172.21.15.18:0/2727981743 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 778200071 0 0) 0x7f278c0c54c0 con 0x7f278c030660 2024-03-29T11:17:56.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.666+0000 7f279159a640 10 monclient(hunting): _negotiate 2024-03-29T11:17:56.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.666+0000 7f279159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:56.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.666+0000 7f279159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:56.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.666+0000 7f279159a640 10 monclient(hunting): my global_id is 4518 2024-03-29T11:17:56.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.666+0000 7f279159a640 1 -- 172.21.15.18:0/2727981743 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f277c003520 con 0x7f278c030660 2024-03-29T11:17:56.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.666+0000 7f279159a640 1 -- 172.21.15.18:0/2727981743 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 856669286 0 0) 0x7f277c003520 con 0x7f278c030660 2024-03-29T11:17:56.645 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.666+0000 7f279159a640 1 -- 172.21.15.18:0/2727981743 >> v1:172.21.15.18:6790/0 conn(0x7f278c038dc0 legacy=0x7f278c0cadd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:17:56.647 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.669+0000 7f279159a640 1 -- 172.21.15.18:0/2727981743 >> v1:172.21.15.70:6789/0 conn(0x7f278c0389e0 legacy=0x7f278c033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:56.647 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.669+0000 7f279159a640 10 monclient: _finish_hunting 0 2024-03-29T11:17:56.647 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.669+0000 7f279159a640 1 monclient: found mon.noname-a 2024-03-29T11:17:56.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.669+0000 7f279159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:56.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.669+0000 7f279159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-29T11:17:56.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.669+0000 7f279159a640 1 -- 172.21.15.18:0/2727981743 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f278c035690 con 0x7f278c030660 2024-03-29T11:17:56.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.669+0000 7f279159a640 10 monclient: _finish_auth 0 2024-03-29T11:17:56.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.669+0000 7f279159a640 10 monclient: _check_auth_tickets 2024-03-29T11:17:56.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.669+0000 7f279159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:56.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.669+0000 7f279159a640 1 -- 172.21.15.18:0/2727981743 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f279404f730 con 0x7f278c030660 2024-03-29T11:17:56.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.669+0000 7f279159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:56.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.669+0000 7f279159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:17:56.648 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.669+0000 7f279159a640 10 monclient: dump: 2024-03-29T11:17:56.648 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:17:56.648 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:56.648 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:56.648 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:56.648 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:56.649 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:56.649 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:17:56.649 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:17:56.649 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:17:56.649 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:56.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.669+0000 7f279159a640 1 -- 172.21.15.18:0/2727981743 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2794075f10 con 0x7f278c030660 2024-03-29T11:17:56.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.669+0000 7f279159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:56.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.669+0000 7f279159a640 1 -- 172.21.15.18:0/2727981743 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f27940758b0 con 0x7f278c030660 2024-03-29T11:17:56.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.669+0000 7f279159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:56.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.669+0000 7f279159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:17:56.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.669+0000 7f279159a640 10 monclient: dump: 2024-03-29T11:17:56.649 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:17:56.649 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:56.649 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:56.649 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:56.649 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:56.650 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:56.650 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:17:56.650 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:17:56.650 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:17:56.650 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:56.650 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.670+0000 7f279359e640 5 monclient: authenticate success, global_id 4518 2024-03-29T11:17:56.650 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.670+0000 7f279359e640 10 monclient: get_monmap_and_config success 2024-03-29T11:17:56.650 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.670+0000 7f279359e640 4 set_mon_vals no callback set 2024-03-29T11:17:56.650 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.670+0000 7f279359e640 10 monclient: shutdown 2024-03-29T11:17:56.650 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.670+0000 7f279359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:56.650 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.670+0000 7f279359e640 1 -- 172.21.15.18:0/2727981743 >> v1:172.21.15.18:6789/0 conn(0x7f278c030660 legacy=0x7f278c0d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:56.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.671+0000 7f279359e640 1 -- 172.21.15.18:0/2727981743 shutdown_connections 2024-03-29T11:17:56.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.671+0000 7f279359e640 1 -- 172.21.15.18:0/2727981743 >> 172.21.15.18:0/2727981743 conn(0x7f278c005c00 msgr2=0x7f278c006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:56.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.671+0000 7f279359e640 1 -- 172.21.15.18:0/2727981743 shutdown_connections 2024-03-29T11:17:56.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.672+0000 7f279359e640 1 -- 172.21.15.18:0/2727981743 wait complete. 2024-03-29T11:17:56.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.673+0000 7f279359e640 10 monclient: build_initial_monmap 2024-03-29T11:17:56.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.673+0000 7f279359e640 10 monclient: monmap: 2024-03-29T11:17:56.653 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:17:56.653 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:17:56.653 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:17:56.653 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:17:56.653 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:17:56.653 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:56.654 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:17:56.654 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:17:56.654 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:17:56.654 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:56.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.673+0000 7f279359e640 1 librados: starting msgr at 2024-03-29T11:17:56.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.673+0000 7f279359e640 1 librados: starting objecter 2024-03-29T11:17:56.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.673+0000 7f279359e640 1 Processor -- start 2024-03-29T11:17:56.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.673+0000 7f279359e640 1 -- start start 2024-03-29T11:17:56.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.673+0000 7f279359e640 1 librados: setting wanted keys 2024-03-29T11:17:56.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.673+0000 7f279359e640 1 librados: calling monclient init 2024-03-29T11:17:56.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.673+0000 7f279359e640 10 monclient: init 2024-03-29T11:17:56.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.674+0000 7f279359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:17:56.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.674+0000 7f279359e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:17:56.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.675+0000 7f279359e640 10 monclient(hunting): picked mon.noname-c con 0x7f278c030660 addr v1:172.21.15.18:6790/0 2024-03-29T11:17:56.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.675+0000 7f279359e640 10 monclient(hunting): picked mon.noname-b con 0x7f278c0389e0 addr v1:172.21.15.70:6789/0 2024-03-29T11:17:56.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.675+0000 7f279359e640 10 monclient(hunting): picked mon.noname-a con 0x7f278c00cd90 addr v1:172.21.15.18:6789/0 2024-03-29T11:17:56.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.675+0000 7f279359e640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f278c00a2d0 con 0x7f278c00cd90 2024-03-29T11:17:56.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.675+0000 7f279359e640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f278c165150 con 0x7f278c0389e0 2024-03-29T11:17:56.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.675+0000 7f279359e640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f278c00a5b0 con 0x7f278c030660 2024-03-29T11:17:56.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.675+0000 7f279359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:17:56.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.675+0000 7f279359e640 10 monclient(hunting): authenticate will time out at 1302.947429s 2024-03-29T11:17:56.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.675+0000 7f2792d9d640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f278c00cd90 0x7f278c16c650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:45570/0 (socket says 172.21.15.18:45570) 2024-03-29T11:17:56.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.675+0000 7f2792d9d640 1 -- 172.21.15.18:0/2900118009 learned_addr learned my addr 172.21.15.18:0/2900118009 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:17:56.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.676+0000 7f2782ffd640 10 client.?.objecter ms_handle_connect 0x7f278c00cd90 2024-03-29T11:17:56.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.676+0000 7f2782ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:56.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.678+0000 7f2782ffd640 10 client.?.objecter ms_handle_connect 0x7f278c030660 2024-03-29T11:17:56.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.678+0000 7f2782ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:56.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.678+0000 7f2782ffd640 1 -- 172.21.15.18:0/2900118009 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2259381219 0 0) 0x7f278c00a5b0 con 0x7f278c030660 2024-03-29T11:17:56.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.678+0000 7f2782ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:17:56.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.678+0000 7f2782ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:56.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.678+0000 7f2782ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:56.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.678+0000 7f2782ffd640 10 monclient(hunting): my global_id is 4598 2024-03-29T11:17:56.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.678+0000 7f2782ffd640 1 -- 172.21.15.18:0/2900118009 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2774004eb0 con 0x7f278c030660 2024-03-29T11:17:56.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.679+0000 7f2782ffd640 1 -- 172.21.15.18:0/2900118009 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2684542118 0 0) 0x7f278c00a2d0 con 0x7f278c00cd90 2024-03-29T11:17:56.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.679+0000 7f2782ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:17:56.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.679+0000 7f2782ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:17:56.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.679+0000 7f2782ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:17:56.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.679+0000 7f2782ffd640 10 monclient(hunting): my global_id is 4521 2024-03-29T11:17:56.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.680+0000 7f2782ffd640 1 -- 172.21.15.18:0/2900118009 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f278c00a5b0 con 0x7f278c00cd90 2024-03-29T11:17:56.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.680+0000 7f2782ffd640 1 -- 172.21.15.18:0/2900118009 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2888920630 0 0) 0x7f2774004eb0 con 0x7f278c030660 2024-03-29T11:17:56.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.680+0000 7f2782ffd640 1 -- 172.21.15.18:0/2900118009 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f278c00a2d0 con 0x7f278c030660 2024-03-29T11:17:56.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.680+0000 7f2782ffd640 10 client.?.objecter ms_handle_connect 0x7f278c0389e0 2024-03-29T11:17:56.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.680+0000 7f2782ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:17:56.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.680+0000 7f2782ffd640 1 -- 172.21.15.18:0/2900118009 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f27840018b0 con 0x7f278c030660 2024-03-29T11:17:56.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.680+0000 7f2782ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:56.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.680+0000 7f2782ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:17:56.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.680+0000 7f2782ffd640 10 monclient(hunting): dump: 2024-03-29T11:17:56.662 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:17:56.662 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:56.662 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:56.662 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:56.662 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:56.662 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:56.663 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:17:56.663 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:17:56.663 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:17:56.663 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:56.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.680+0000 7f2782ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:17:56.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.680+0000 7f2782ffd640 1 -- 172.21.15.18:0/2900118009 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4044220481 0 0) 0x7f278c00a5b0 con 0x7f278c00cd90 2024-03-29T11:17:56.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.680+0000 7f2782ffd640 1 -- 172.21.15.18:0/2900118009 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2774004eb0 con 0x7f278c00cd90 2024-03-29T11:17:56.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.681+0000 7f2782ffd640 1 -- 172.21.15.18:0/2900118009 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1108164894 0 0) 0x7f278c00a2d0 con 0x7f278c030660 2024-03-29T11:17:56.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.681+0000 7f2782ffd640 1 -- 172.21.15.18:0/2900118009 >> v1:172.21.15.70:6789/0 conn(0x7f278c0389e0 legacy=0x7f278c00bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:56.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.681+0000 7f2782ffd640 1 -- 172.21.15.18:0/2900118009 >> v1:172.21.15.18:6789/0 conn(0x7f278c00cd90 legacy=0x7f278c16c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:56.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.681+0000 7f2782ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:17:56.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.681+0000 7f2782ffd640 1 monclient: found mon.c 2024-03-29T11:17:56.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.681+0000 7f2782ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:17:56.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.681+0000 7f2782ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:17:56.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.681+0000 7f2782ffd640 1 -- 172.21.15.18:0/2900118009 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f278c164520 con 0x7f278c030660 2024-03-29T11:17:56.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.681+0000 7f2782ffd640 10 monclient: _finish_auth 0 2024-03-29T11:17:56.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.681+0000 7f2782ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:17:56.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.681+0000 7f2782ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:17:56.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.681+0000 7f2782ffd640 1 -- 172.21.15.18:0/2900118009 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2784001a80 con 0x7f278c030660 2024-03-29T11:17:56.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.681+0000 7f279359e640 5 monclient: authenticate success, global_id 4598 2024-03-29T11:17:56.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.681+0000 7f279359e640 10 monclient: _renew_subs 2024-03-29T11:17:56.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.681+0000 7f279359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:17:56.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.681+0000 7f279359e640 1 -- 172.21.15.18:0/2900118009 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f278c0c81d0 con 0x7f278c030660 2024-03-29T11:17:56.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.682+0000 7f279359e640 10 client.4598.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:17:56.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.682+0000 7f279359e640 10 monclient: _renew_subs 2024-03-29T11:17:56.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.682+0000 7f279359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:17:56.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.682+0000 7f279359e640 1 -- 172.21.15.18:0/2900118009 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f278c160e10 con 0x7f278c030660 2024-03-29T11:17:56.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.682+0000 7f279359e640 1 librados: init done 2024-03-29T11:17:56.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.682+0000 7f2782ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:17:56.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.682+0000 7f2782ffd640 1 -- 172.21.15.18:0/2900118009 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f27840038c0 con 0x7f278c030660 2024-03-29T11:17:56.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.682+0000 7f2782ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:17:56.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.682+0000 7f2782ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:17:56.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.682+0000 7f2782ffd640 10 monclient: dump: 2024-03-29T11:17:56.665 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:17:56.665 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:17:56.665 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:56.665 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:17:56.665 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:17:56.665 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:17:56.665 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:17:56.665 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:17:56.665 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:17:56.665 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:17:56.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.683+0000 7f2783fff640 4 set_mon_vals no callback set 2024-03-29T11:17:56.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.684+0000 7f279359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:56.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.684+0000 7f279359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:56.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.684+0000 7f279359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:17:56.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.684+0000 7f279359e640 1 -- 172.21.15.18:0/2900118009 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2778005080 con 0x7f278c030660 2024-03-29T11:17:56.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.686+0000 7f2782ffd640 1 -- 172.21.15.18:0/2900118009 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f2784019890 con 0x7f278c030660 2024-03-29T11:17:56.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.687+0000 7f2782ffd640 1 -- 172.21.15.18:0/2900118009 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f2784083770 con 0x7f278c030660 2024-03-29T11:17:56.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.687+0000 7f2782ffd640 10 client.4598.objecter ms_dispatch 0x7f2794000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:17:56.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.687+0000 7f2782ffd640 3 client.4598.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:17:56.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.687+0000 7f2782ffd640 3 client.4598.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:17:56.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.687+0000 7f2782ffd640 20 client.4598.objecter dump_active .. 0 homeless 2024-03-29T11:17:56.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.689+0000 7f2791d9b640 10 monclient: get_auth_request con 0x7f277406dce0 auth_method 0 2024-03-29T11:17:56.668 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.690+0000 7f2782ffd640 10 client.4598.objecter ms_handle_connect 0x7f277406dce0 2024-03-29T11:17:56.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.695+0000 7f2782ffd640 1 -- 172.21.15.18:0/2900118009 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f278404f050 con 0x7f278c030660 2024-03-29T11:17:56.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.695+0000 7f2782ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:17:56.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:56.695+0000 7f2782ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:17:57.001 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:57.024+0000 7f279359e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:57.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:57.024+0000 7f279359e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:57.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:57.024+0000 7f279359e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:17:57.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:57.024+0000 7f279359e640 1 -- 172.21.15.18:0/2900118009 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2778005330 con 0x7f278c030660 2024-03-29T11:17:57.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:57.025+0000 7f2782ffd640 1 -- 172.21.15.18:0/2900118009 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f2784056ba0 con 0x7f278c030660 2024-03-29T11:17:57.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:57.025+0000 7f2782ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:17:57.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:57.025+0000 7f2782ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:17:57.003 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:17:57.003 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:17:57.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:57.029+0000 7f279359e640 10 librados: watch_flush enter 2024-03-29T11:17:57.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:57.029+0000 7f279359e640 10 librados: watch_flush exit 2024-03-29T11:17:57.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:57.029+0000 7f279359e640 20 client.4598.objecter shutdown clearing up homeless session... 2024-03-29T11:17:57.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:57.029+0000 7f279359e640 10 client.4598.objecter successfully canceled tick 2024-03-29T11:17:57.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:57.030+0000 7f279359e640 1 -- 172.21.15.18:0/2900118009 >> v1:172.21.15.70:6816/3101871562 conn(0x7f277406dce0 legacy=0x7f27740701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:57.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:57.030+0000 7f279359e640 10 monclient: shutdown 2024-03-29T11:17:57.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:57.030+0000 7f279359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:17:57.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:57.030+0000 7f279359e640 1 -- 172.21.15.18:0/2900118009 >> v1:172.21.15.18:6790/0 conn(0x7f278c030660 legacy=0x7f278c164a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:17:57.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:57.030+0000 7f279359e640 1 -- 172.21.15.18:0/2900118009 shutdown_connections 2024-03-29T11:17:57.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:57.030+0000 7f279359e640 1 -- 172.21.15.18:0/2900118009 >> 172.21.15.18:0/2900118009 conn(0x7f278c005c00 msgr2=0x7f278c005fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:17:57.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:57.030+0000 7f279359e640 1 -- 172.21.15.18:0/2900118009 shutdown_connections 2024-03-29T11:17:57.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:57.030+0000 7f279359e640 1 -- 172.21.15.18:0/2900118009 wait complete. 2024-03-29T11:17:57.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:17:57.030+0000 7f279359e640 1 librados: shutdown 2024-03-29T11:17:57.025 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:03.833 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:03.991 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.014+0000 7fd494b17640 10 monclient: get_monmap_and_config 2024-03-29T11:18:03.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.014+0000 7fd494b17640 10 monclient: build_initial_monmap 2024-03-29T11:18:03.992 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.014+0000 7fd494b17640 10 monclient: monmap: 2024-03-29T11:18:03.992 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:18:03.992 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:03.992 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:18:04.015657+0000 2024-03-29T11:18:03.992 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:18:04.015657+0000 2024-03-29T11:18:03.992 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:03.992 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:03.992 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:18:03.992 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:18:03.992 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:18:03.992 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:03.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.015+0000 7fd494b17640 1 Processor -- start 2024-03-29T11:18:03.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.016+0000 7fd494b17640 1 -- start start 2024-03-29T11:18:03.993 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.016+0000 7fd494b17640 10 monclient: init 2024-03-29T11:18:03.994 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.017+0000 7fd494b17640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:03.994 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.017+0000 7fd494b17640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:18:03.994 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.017+0000 7fd494b17640 10 monclient(hunting): picked mon.noname-a con 0x7fd4880389e0 addr v1:172.21.15.18:6789/0 2024-03-29T11:18:03.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.018+0000 7fd494b17640 10 monclient(hunting): picked mon.noname-c con 0x7fd488038dc0 addr v1:172.21.15.70:6789/0 2024-03-29T11:18:03.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.018+0000 7fd494b17640 10 monclient(hunting): picked mon.noname-b con 0x7fd488030660 addr v1:172.21.15.18:6790/0 2024-03-29T11:18:03.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.018+0000 7fd494b17640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4880c54c0 con 0x7fd4880389e0 2024-03-29T11:18:03.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.018+0000 7fd494b17640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd488036430 con 0x7fd488038dc0 2024-03-29T11:18:03.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.018+0000 7fd494b17640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd488039840 con 0x7fd488030660 2024-03-29T11:18:03.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.018+0000 7fd494b17640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:03.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.018+0000 7fd494b17640 10 monclient(hunting): authenticate will time out at 1310.290605s 2024-03-29T11:18:03.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.021+0000 7fd48f7fe640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fd4880389e0 0x7fd488033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:60684/0 (socket says 172.21.15.18:60684) 2024-03-29T11:18:03.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.021+0000 7fd48f7fe640 1 -- 172.21.15.18:0/1588314644 learned_addr learned my addr 172.21.15.18:0/1588314644 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:18:04.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.022+0000 7fd48e7fc640 1 -- 172.21.15.18:0/1588314644 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3771206470 0 0) 0x7fd488039840 con 0x7fd488030660 2024-03-29T11:18:04.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.022+0000 7fd48e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:04.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.023+0000 7fd48e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:04.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.023+0000 7fd48e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:04.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.023+0000 7fd48e7fc640 10 monclient(hunting): my global_id is 4601 2024-03-29T11:18:04.002 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.023+0000 7fd48e7fc640 1 -- 172.21.15.18:0/1588314644 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd480003520 con 0x7fd488030660 2024-03-29T11:18:04.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.023+0000 7fd48e7fc640 1 -- 172.21.15.18:0/1588314644 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3534597998 0 0) 0x7fd480003520 con 0x7fd488030660 2024-03-29T11:18:04.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.023+0000 7fd48e7fc640 1 -- 172.21.15.18:0/1588314644 >> v1:172.21.15.70:6789/0 conn(0x7fd488038dc0 legacy=0x7fd4880cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:18:04.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.023+0000 7fd48e7fc640 1 -- 172.21.15.18:0/1588314644 >> v1:172.21.15.18:6789/0 conn(0x7fd4880389e0 legacy=0x7fd488033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:18:04.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.023+0000 7fd48e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:18:04.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.023+0000 7fd48e7fc640 1 monclient: found mon.noname-b 2024-03-29T11:18:04.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.023+0000 7fd48e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:04.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.023+0000 7fd48e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:18:04.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.023+0000 7fd48e7fc640 1 -- 172.21.15.18:0/1588314644 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd488035690 con 0x7fd488030660 2024-03-29T11:18:04.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.023+0000 7fd48e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:18:04.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.024+0000 7fd48e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:18:04.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.024+0000 7fd48e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:04.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.024+0000 7fd48e7fc640 1 -- 172.21.15.18:0/1588314644 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fd49004f730 con 0x7fd488030660 2024-03-29T11:18:04.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.024+0000 7fd48e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:04.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.024+0000 7fd48e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:04.003 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.024+0000 7fd48e7fc640 10 monclient: dump: 2024-03-29T11:18:04.004 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:18:04.004 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:04.004 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:04.004 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:04.004 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:04.004 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:04.004 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:18:04.004 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:18:04.004 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:18:04.004 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:04.004 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.024+0000 7fd48e7fc640 1 -- 172.21.15.18:0/1588314644 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd4900008d0 con 0x7fd488030660 2024-03-29T11:18:04.004 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.024+0000 7fd48e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:04.004 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.024+0000 7fd48e7fc640 1 -- 172.21.15.18:0/1588314644 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fd4900758b0 con 0x7fd488030660 2024-03-29T11:18:04.004 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.024+0000 7fd48e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:04.004 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.024+0000 7fd48e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:04.004 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.024+0000 7fd48e7fc640 10 monclient: dump: 2024-03-29T11:18:04.005 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:18:04.005 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:04.005 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:04.005 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:04.005 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:04.005 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:04.005 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:18:04.005 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:18:04.005 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:18:04.005 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:04.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.024+0000 7fd494b17640 5 monclient: authenticate success, global_id 4601 2024-03-29T11:18:04.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.025+0000 7fd494b17640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:04.005 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.025+0000 7fd494b17640 4 set_mon_vals no callback set 2024-03-29T11:18:04.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.026+0000 7fd494b17640 10 monclient: shutdown 2024-03-29T11:18:04.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.026+0000 7fd494b17640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:04.006 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.026+0000 7fd494b17640 1 -- 172.21.15.18:0/1588314644 >> v1:172.21.15.18:6790/0 conn(0x7fd488030660 legacy=0x7fd4880d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:04.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.029+0000 7fd494b17640 1 -- 172.21.15.18:0/1588314644 shutdown_connections 2024-03-29T11:18:04.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.030+0000 7fd494b17640 1 -- 172.21.15.18:0/1588314644 >> 172.21.15.18:0/1588314644 conn(0x7fd488005c00 msgr2=0x7fd488006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:04.007 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.030+0000 7fd494b17640 1 -- 172.21.15.18:0/1588314644 shutdown_connections 2024-03-29T11:18:04.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.031+0000 7fd494b17640 1 -- 172.21.15.18:0/1588314644 wait complete. 2024-03-29T11:18:04.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.032+0000 7fd494b17640 10 monclient: build_initial_monmap 2024-03-29T11:18:04.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.032+0000 7fd494b17640 10 monclient: monmap: 2024-03-29T11:18:04.010 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:18:04.010 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:04.010 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:18:04.010 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:18:04.011 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:04.011 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:04.011 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:18:04.011 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:18:04.011 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:18:04.011 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:04.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.032+0000 7fd494b17640 1 librados: starting msgr at 2024-03-29T11:18:04.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.032+0000 7fd494b17640 1 librados: starting objecter 2024-03-29T11:18:04.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.033+0000 7fd494b17640 1 Processor -- start 2024-03-29T11:18:04.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.033+0000 7fd494b17640 1 -- start start 2024-03-29T11:18:04.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.033+0000 7fd494b17640 1 librados: setting wanted keys 2024-03-29T11:18:04.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.033+0000 7fd494b17640 1 librados: calling monclient init 2024-03-29T11:18:04.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.033+0000 7fd494b17640 10 monclient: init 2024-03-29T11:18:04.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.034+0000 7fd494b17640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:04.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.034+0000 7fd494b17640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:18:04.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.037+0000 7fd494b17640 10 monclient(hunting): picked mon.noname-b con 0x7fd488030660 addr v1:172.21.15.70:6789/0 2024-03-29T11:18:04.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.038+0000 7fd494b17640 10 monclient(hunting): picked mon.noname-a con 0x7fd4880389e0 addr v1:172.21.15.18:6789/0 2024-03-29T11:18:04.015 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.038+0000 7fd48effd640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7fd4880389e0 0x7fd48815f9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:33080/0 (socket says 172.21.15.18:33080) 2024-03-29T11:18:04.017 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.040+0000 7fd494b17640 10 monclient(hunting): picked mon.noname-c con 0x7fd48815cbb0 addr v1:172.21.15.18:6790/0 2024-03-29T11:18:04.017 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.040+0000 7fd48effd640 1 -- 172.21.15.18:0/3277621832 learned_addr learned my addr 172.21.15.18:0/3277621832 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:18:04.017 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.040+0000 7fd494b17640 1 -- 172.21.15.18:0/3277621832 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4880ca360 con 0x7fd4880389e0 2024-03-29T11:18:04.017 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.040+0000 7fd494b17640 1 -- 172.21.15.18:0/3277621832 --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4880ca4f0 con 0x7fd488030660 2024-03-29T11:18:04.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.040+0000 7fd494b17640 1 -- 172.21.15.18:0/3277621832 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4880ca680 con 0x7fd48815cbb0 2024-03-29T11:18:04.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.040+0000 7fd494b17640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:04.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.040+0000 7fd494b17640 10 monclient(hunting): authenticate will time out at 1310.312201s 2024-03-29T11:18:04.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.041+0000 7fd47bfff640 10 client.?.objecter ms_handle_connect 0x7fd488030660 2024-03-29T11:18:04.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.041+0000 7fd47bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:04.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.041+0000 7fd47bfff640 10 client.?.objecter ms_handle_connect 0x7fd4880389e0 2024-03-29T11:18:04.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.041+0000 7fd47bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:04.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.041+0000 7fd47bfff640 1 -- 172.21.15.18:0/3277621832 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1695452719 0 0) 0x7fd4880ca360 con 0x7fd4880389e0 2024-03-29T11:18:04.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.041+0000 7fd47bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:04.022 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.041+0000 7fd47bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:04.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.041+0000 7fd47bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:04.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.041+0000 7fd47bfff640 10 monclient(hunting): my global_id is 4524 2024-03-29T11:18:04.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.041+0000 7fd47bfff640 1 -- 172.21.15.18:0/3277621832 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd474004eb0 con 0x7fd4880389e0 2024-03-29T11:18:04.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.041+0000 7fd47bfff640 1 -- 172.21.15.18:0/3277621832 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1775328466 0 0) 0x7fd4880ca4f0 con 0x7fd488030660 2024-03-29T11:18:04.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.041+0000 7fd47bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:04.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.041+0000 7fd47bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:04.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.041+0000 7fd47bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:04.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.041+0000 7fd47bfff640 10 monclient(hunting): my global_id is 4489 2024-03-29T11:18:04.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.041+0000 7fd47bfff640 1 -- 172.21.15.18:0/3277621832 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4880ca360 con 0x7fd488030660 2024-03-29T11:18:04.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.042+0000 7fd47bfff640 1 -- 172.21.15.18:0/3277621832 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2996550037 0 0) 0x7fd474004eb0 con 0x7fd4880389e0 2024-03-29T11:18:04.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.042+0000 7fd47bfff640 1 -- 172.21.15.18:0/3277621832 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd4880ca4f0 con 0x7fd4880389e0 2024-03-29T11:18:04.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.042+0000 7fd47bfff640 1 -- 172.21.15.18:0/3277621832 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fd47c003f10 con 0x7fd4880389e0 2024-03-29T11:18:04.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.042+0000 7fd47bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:04.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.042+0000 7fd47bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:18:04.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.042+0000 7fd47bfff640 10 monclient(hunting): dump: 2024-03-29T11:18:04.024 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:18:04.024 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:04.024 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:04.024 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:04.024 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:04.024 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:04.024 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:18:04.024 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:18:04.024 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:18:04.024 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:04.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.042+0000 7fd47bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:04.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.042+0000 7fd47bfff640 1 -- 172.21.15.18:0/3277621832 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2865433622 0 0) 0x7fd4880ca4f0 con 0x7fd4880389e0 2024-03-29T11:18:04.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.042+0000 7fd47bfff640 1 -- 172.21.15.18:0/3277621832 >> v1:172.21.15.18:6790/0 conn(0x7fd48815cbb0 legacy=0x7fd4881600f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:04.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.042+0000 7fd47bfff640 1 -- 172.21.15.18:0/3277621832 >> v1:172.21.15.70:6789/0 conn(0x7fd488030660 legacy=0x7fd48815c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:04.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.042+0000 7fd47bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:18:04.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.042+0000 7fd47bfff640 1 monclient: found mon.a 2024-03-29T11:18:04.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.042+0000 7fd47bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:04.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.042+0000 7fd47bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:18:04.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.042+0000 7fd47bfff640 1 -- 172.21.15.18:0/3277621832 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd48816b030 con 0x7fd4880389e0 2024-03-29T11:18:04.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.043+0000 7fd47bfff640 10 monclient: _finish_auth 0 2024-03-29T11:18:04.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.043+0000 7fd47bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:18:04.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.043+0000 7fd47bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:04.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.043+0000 7fd47bfff640 10 client.?.objecter ms_handle_connect 0x7fd48815cbb0 2024-03-29T11:18:04.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.043+0000 7fd494b17640 5 monclient: authenticate success, global_id 4524 2024-03-29T11:18:04.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.043+0000 7fd494b17640 10 monclient: _renew_subs 2024-03-29T11:18:04.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.043+0000 7fd47bfff640 10 client.4524.objecter resend_mon_ops 2024-03-29T11:18:04.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.043+0000 7fd47bfff640 1 -- 172.21.15.18:0/3277621832 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd47c004420 con 0x7fd4880389e0 2024-03-29T11:18:04.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.043+0000 7fd494b17640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:18:04.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.043+0000 7fd494b17640 1 -- 172.21.15.18:0/3277621832 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd488168000 con 0x7fd4880389e0 2024-03-29T11:18:04.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.043+0000 7fd494b17640 10 client.4524.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:04.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.043+0000 7fd47bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:04.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.043+0000 7fd48dffb640 4 set_mon_vals no callback set 2024-03-29T11:18:04.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.044+0000 7fd47bfff640 1 -- 172.21.15.18:0/3277621832 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fd47c001070 con 0x7fd4880389e0 2024-03-29T11:18:04.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.044+0000 7fd47bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:04.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.044+0000 7fd47bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:04.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.044+0000 7fd47bfff640 10 monclient: dump: 2024-03-29T11:18:04.026 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:18:04.026 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:04.026 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:04.026 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:04.026 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:04.026 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:04.026 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:18:04.027 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:18:04.027 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:18:04.027 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:04.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.044+0000 7fd494b17640 10 monclient: _renew_subs 2024-03-29T11:18:04.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.044+0000 7fd494b17640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:18:04.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.044+0000 7fd494b17640 1 -- 172.21.15.18:0/3277621832 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd488168530 con 0x7fd4880389e0 2024-03-29T11:18:04.027 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.044+0000 7fd494b17640 1 librados: init done 2024-03-29T11:18:04.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.046+0000 7fd47bfff640 1 -- 172.21.15.18:0/3277621832 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7fd47c001290 con 0x7fd4880389e0 2024-03-29T11:18:04.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.046+0000 7fd494b17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:04.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.046+0000 7fd494b17640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:04.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.046+0000 7fd494b17640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:18:04.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.046+0000 7fd494b17640 1 -- 172.21.15.18:0/3277621832 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd488007b70 con 0x7fd4880389e0 2024-03-29T11:18:04.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.047+0000 7fd48f7fe640 10 monclient: get_auth_request con 0x7fd47406dcb0 auth_method 0 2024-03-29T11:18:04.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.047+0000 7fd47bfff640 1 -- 172.21.15.18:0/3277621832 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7fd47c085990 con 0x7fd4880389e0 2024-03-29T11:18:04.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.047+0000 7fd47bfff640 10 client.4524.objecter ms_dispatch 0x7fd490000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:04.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.047+0000 7fd47bfff640 3 client.4524.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:04.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.047+0000 7fd47bfff640 3 client.4524.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:04.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.047+0000 7fd47bfff640 20 client.4524.objecter dump_active .. 0 homeless 2024-03-29T11:18:04.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.047+0000 7fd47bfff640 10 client.4524.objecter ms_handle_connect 0x7fd47406dcb0 2024-03-29T11:18:04.031 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.054+0000 7fd47bfff640 1 -- 172.21.15.18:0/3277621832 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd47c050240 con 0x7fd4880389e0 2024-03-29T11:18:04.031 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.054+0000 7fd47bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:04.031 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.054+0000 7fd47bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:04.364 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.387+0000 7fd494b17640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:04.364 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.387+0000 7fd494b17640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:04.365 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.387+0000 7fd494b17640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:18:04.365 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.387+0000 7fd494b17640 1 -- 172.21.15.18:0/3277621832 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd488005120 con 0x7fd4880389e0 2024-03-29T11:18:04.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.390+0000 7fd47bfff640 1 -- 172.21.15.18:0/3277621832 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7fd47c052170 con 0x7fd4880389e0 2024-03-29T11:18:04.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.390+0000 7fd47bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:04.367 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.390+0000 7fd47bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:04.368 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:18:04.368 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:04.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.395+0000 7fd494b17640 10 librados: watch_flush enter 2024-03-29T11:18:04.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.396+0000 7fd494b17640 10 librados: watch_flush exit 2024-03-29T11:18:04.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.396+0000 7fd494b17640 20 client.4524.objecter shutdown clearing up homeless session... 2024-03-29T11:18:04.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.396+0000 7fd494b17640 10 client.4524.objecter successfully canceled tick 2024-03-29T11:18:04.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.396+0000 7fd494b17640 1 -- 172.21.15.18:0/3277621832 >> v1:172.21.15.70:6816/3101871562 conn(0x7fd47406dcb0 legacy=0x7fd474070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:04.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.396+0000 7fd494b17640 10 monclient: shutdown 2024-03-29T11:18:04.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.396+0000 7fd494b17640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:04.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.396+0000 7fd494b17640 1 -- 172.21.15.18:0/3277621832 >> v1:172.21.15.18:6789/0 conn(0x7fd4880389e0 legacy=0x7fd48815f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:04.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.396+0000 7fd494b17640 1 -- 172.21.15.18:0/3277621832 shutdown_connections 2024-03-29T11:18:04.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.396+0000 7fd494b17640 1 -- 172.21.15.18:0/3277621832 >> 172.21.15.18:0/3277621832 conn(0x7fd488005c00 msgr2=0x7fd488032420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:04.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.397+0000 7fd494b17640 1 -- 172.21.15.18:0/3277621832 shutdown_connections 2024-03-29T11:18:04.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.397+0000 7fd494b17640 1 -- 172.21.15.18:0/3277621832 wait complete. 2024-03-29T11:18:04.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:04.397+0000 7fd494b17640 1 librados: shutdown 2024-03-29T11:18:04.390 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:11.197 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:11.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.352+0000 7f9cb3274640 10 monclient: get_monmap_and_config 2024-03-29T11:18:11.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.352+0000 7f9cb3274640 10 monclient: build_initial_monmap 2024-03-29T11:18:11.330 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.352+0000 7f9cb3274640 10 monclient: monmap: 2024-03-29T11:18:11.330 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:18:11.330 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:11.330 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:18:11.353874+0000 2024-03-29T11:18:11.330 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:18:11.353874+0000 2024-03-29T11:18:11.331 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:11.331 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:11.331 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:18:11.331 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:18:11.331 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:18:11.331 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:11.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.354+0000 7f9cb3274640 1 Processor -- start 2024-03-29T11:18:11.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.354+0000 7f9cb3274640 1 -- start start 2024-03-29T11:18:11.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.354+0000 7f9cb3274640 10 monclient: init 2024-03-29T11:18:11.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.355+0000 7f9cb3274640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:11.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.355+0000 7f9cb3274640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:18:11.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.355+0000 7f9cb3274640 10 monclient(hunting): picked mon.noname-b con 0x7f9cac0b1f60 addr v1:172.21.15.18:6790/0 2024-03-29T11:18:11.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.355+0000 7f9cb3274640 10 monclient(hunting): picked mon.noname-a con 0x7f9cac071eb0 addr v1:172.21.15.18:6789/0 2024-03-29T11:18:11.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.355+0000 7f9cb1010640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f9cac0b1f60 0x7f9cac143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:53926/0 (socket says 172.21.15.18:53926) 2024-03-29T11:18:11.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.355+0000 7f9cb3274640 10 monclient(hunting): picked mon.noname-c con 0x7f9cac143d10 addr v1:172.21.15.70:6789/0 2024-03-29T11:18:11.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.356+0000 7f9cb1010640 1 -- 172.21.15.18:0/190632344 learned_addr learned my addr 172.21.15.18:0/190632344 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:18:11.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.356+0000 7f9cb3274640 1 -- 172.21.15.18:0/190632344 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cac06ef80 con 0x7f9cac071eb0 2024-03-29T11:18:11.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.356+0000 7f9cb3274640 1 -- 172.21.15.18:0/190632344 --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cac06f260 con 0x7f9cac143d10 2024-03-29T11:18:11.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.356+0000 7f9cb3274640 1 -- 172.21.15.18:0/190632344 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cac0b40c0 con 0x7f9cac0b1f60 2024-03-29T11:18:11.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.356+0000 7f9cb3274640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:11.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.356+0000 7f9cb3274640 10 monclient(hunting): authenticate will time out at 1317.628145s 2024-03-29T11:18:11.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.356+0000 7f9ca3fff640 1 -- 172.21.15.18:0/190632344 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1627191035 0 0) 0x7f9cac06f260 con 0x7f9cac143d10 2024-03-29T11:18:11.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.356+0000 7f9ca3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:11.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.356+0000 7f9ca3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:11.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.356+0000 7f9ca3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:11.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.356+0000 7f9ca3fff640 10 monclient(hunting): my global_id is 4492 2024-03-29T11:18:11.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.356+0000 7f9ca3fff640 1 -- 172.21.15.18:0/190632344 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c98003520 con 0x7f9cac143d10 2024-03-29T11:18:11.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.356+0000 7f9ca3fff640 1 -- 172.21.15.18:0/190632344 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3861175113 0 0) 0x7f9cac0b40c0 con 0x7f9cac0b1f60 2024-03-29T11:18:11.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.356+0000 7f9ca3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:11.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.356+0000 7f9ca3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:11.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.356+0000 7f9ca3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:11.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.356+0000 7f9ca3fff640 10 monclient(hunting): my global_id is 4607 2024-03-29T11:18:11.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.356+0000 7f9ca3fff640 1 -- 172.21.15.18:0/190632344 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cac06f260 con 0x7f9cac0b1f60 2024-03-29T11:18:11.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.357+0000 7f9ca3fff640 1 -- 172.21.15.18:0/190632344 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2694791140 0 0) 0x7f9cac06f260 con 0x7f9cac0b1f60 2024-03-29T11:18:11.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.357+0000 7f9ca3fff640 1 -- 172.21.15.18:0/190632344 >> v1:172.21.15.70:6789/0 conn(0x7f9cac143d10 legacy=0x7f9cac146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:11.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.357+0000 7f9ca3fff640 1 -- 172.21.15.18:0/190632344 >> v1:172.21.15.18:6789/0 conn(0x7f9cac071eb0 legacy=0x7f9cac146470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:18:11.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.359+0000 7f9ca3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:18:11.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.359+0000 7f9ca3fff640 1 monclient: found mon.noname-b 2024-03-29T11:18:11.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.359+0000 7f9ca3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:11.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.359+0000 7f9ca3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:18:11.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.359+0000 7f9ca3fff640 1 -- 172.21.15.18:0/190632344 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cac154d60 con 0x7f9cac0b1f60 2024-03-29T11:18:11.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.359+0000 7f9ca3fff640 10 monclient: _finish_auth 0 2024-03-29T11:18:11.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.359+0000 7f9ca3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:18:11.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.359+0000 7f9ca3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:11.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.359+0000 7f9ca3fff640 1 -- 172.21.15.18:0/190632344 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f9ca8001630 con 0x7f9cac0b1f60 2024-03-29T11:18:11.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.359+0000 7f9ca3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:11.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.359+0000 7f9ca3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:11.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.359+0000 7f9ca3fff640 10 monclient: dump: 2024-03-29T11:18:11.339 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:18:11.340 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:11.340 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:11.340 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:11.340 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:11.340 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:11.340 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:18:11.340 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:18:11.340 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:18:11.340 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:11.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.359+0000 7f9cb3274640 5 monclient: authenticate success, global_id 4607 2024-03-29T11:18:11.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.359+0000 7f9cb3274640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:11.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.359+0000 7f9ca3fff640 1 -- 172.21.15.18:0/190632344 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ca8001880 con 0x7f9cac0b1f60 2024-03-29T11:18:11.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.359+0000 7f9ca3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:11.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.359+0000 7f9ca3fff640 1 -- 172.21.15.18:0/190632344 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f9ca8001c10 con 0x7f9cac0b1f60 2024-03-29T11:18:11.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.359+0000 7f9ca3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:11.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.359+0000 7f9ca3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:11.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.359+0000 7f9ca3fff640 10 monclient: dump: 2024-03-29T11:18:11.341 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:18:11.341 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:11.341 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:11.341 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:11.341 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:11.341 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:11.341 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:18:11.341 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:18:11.341 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:18:11.341 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:11.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.359+0000 7f9cb3274640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:11.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.359+0000 7f9cb3274640 4 set_mon_vals no callback set 2024-03-29T11:18:11.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.360+0000 7f9cb3274640 10 monclient: shutdown 2024-03-29T11:18:11.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.360+0000 7f9cb3274640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:11.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.360+0000 7f9cb3274640 1 -- 172.21.15.18:0/190632344 >> v1:172.21.15.18:6790/0 conn(0x7f9cac0b1f60 legacy=0x7f9cac143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:11.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.360+0000 7f9cb3274640 1 -- 172.21.15.18:0/190632344 shutdown_connections 2024-03-29T11:18:11.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.360+0000 7f9cb3274640 1 -- 172.21.15.18:0/190632344 >> 172.21.15.18:0/190632344 conn(0x7f9cac066a00 msgr2=0x7f9cac066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:11.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.360+0000 7f9cb3274640 1 -- 172.21.15.18:0/190632344 shutdown_connections 2024-03-29T11:18:11.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.360+0000 7f9cb3274640 1 -- 172.21.15.18:0/190632344 wait complete. 2024-03-29T11:18:11.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.361+0000 7f9cb3274640 10 monclient: build_initial_monmap 2024-03-29T11:18:11.342 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.361+0000 7f9cb3274640 10 monclient: monmap: 2024-03-29T11:18:11.342 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:18:11.342 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:11.342 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:18:11.343 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:18:11.343 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:11.343 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:11.343 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:18:11.343 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:18:11.343 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:18:11.343 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:11.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.361+0000 7f9cb3274640 1 librados: starting msgr at 2024-03-29T11:18:11.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.361+0000 7f9cb3274640 1 librados: starting objecter 2024-03-29T11:18:11.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.361+0000 7f9cb3274640 1 Processor -- start 2024-03-29T11:18:11.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.361+0000 7f9cb3274640 1 -- start start 2024-03-29T11:18:11.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.361+0000 7f9cb3274640 1 librados: setting wanted keys 2024-03-29T11:18:11.343 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.361+0000 7f9cb3274640 1 librados: calling monclient init 2024-03-29T11:18:11.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.361+0000 7f9cb3274640 10 monclient: init 2024-03-29T11:18:11.349 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.362+0000 7f9cb3274640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:11.349 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.362+0000 7f9cb3274640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:18:11.349 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.362+0000 7f9cb3274640 10 monclient(hunting): picked mon.noname-c con 0x7f9cac0b1f60 addr v1:172.21.15.18:6790/0 2024-03-29T11:18:11.349 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.362+0000 7f9cb3274640 10 monclient(hunting): picked mon.noname-b con 0x7f9cac143d10 addr v1:172.21.15.70:6789/0 2024-03-29T11:18:11.349 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.364+0000 7f9cb3274640 10 monclient(hunting): picked mon.noname-a con 0x7f9cac141a00 addr v1:172.21.15.18:6789/0 2024-03-29T11:18:11.349 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.364+0000 7f9cb3274640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cac1420f0 con 0x7f9cac141a00 2024-03-29T11:18:11.350 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.364+0000 7f9cb3274640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cac133750 con 0x7f9cac143d10 2024-03-29T11:18:11.350 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.364+0000 7f9cb3274640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cac1357f0 con 0x7f9cac0b1f60 2024-03-29T11:18:11.350 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.364+0000 7f9cb3274640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:11.350 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.364+0000 7f9cb3274640 10 monclient(hunting): authenticate will time out at 1317.636078s 2024-03-29T11:18:11.350 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.364+0000 7f9cb1010640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f9cac0b1f60 0x7f9cac141090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:57244/0 (socket says 172.21.15.18:57244) 2024-03-29T11:18:11.350 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.364+0000 7f9cb1010640 1 -- 172.21.15.18:0/2572049264 learned_addr learned my addr 172.21.15.18:0/2572049264 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:18:11.350 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.366+0000 7f9ca1ffb640 10 client.?.objecter ms_handle_connect 0x7f9cac0b1f60 2024-03-29T11:18:11.350 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.366+0000 7f9ca1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:11.350 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.366+0000 7f9ca1ffb640 10 client.?.objecter ms_handle_connect 0x7f9cac141a00 2024-03-29T11:18:11.350 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.366+0000 7f9ca1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:11.350 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.366+0000 7f9ca1ffb640 10 client.?.objecter ms_handle_connect 0x7f9cac143d10 2024-03-29T11:18:11.350 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.366+0000 7f9ca1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:11.350 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.366+0000 7f9ca1ffb640 1 -- 172.21.15.18:0/2572049264 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4092711629 0 0) 0x7f9cac1357f0 con 0x7f9cac0b1f60 2024-03-29T11:18:11.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.366+0000 7f9ca1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:11.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.366+0000 7f9ca1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:11.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.366+0000 7f9ca1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:11.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.366+0000 7f9ca1ffb640 10 monclient(hunting): my global_id is 4610 2024-03-29T11:18:11.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.366+0000 7f9ca1ffb640 1 -- 172.21.15.18:0/2572049264 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c90004eb0 con 0x7f9cac0b1f60 2024-03-29T11:18:11.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.366+0000 7f9ca1ffb640 1 -- 172.21.15.18:0/2572049264 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1028311147 0 0) 0x7f9cac1420f0 con 0x7f9cac141a00 2024-03-29T11:18:11.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.366+0000 7f9ca1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:11.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.366+0000 7f9ca1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:11.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.366+0000 7f9ca1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:11.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.366+0000 7f9ca1ffb640 10 monclient(hunting): my global_id is 4527 2024-03-29T11:18:11.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.366+0000 7f9ca1ffb640 1 -- 172.21.15.18:0/2572049264 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cac1357f0 con 0x7f9cac141a00 2024-03-29T11:18:11.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.366+0000 7f9ca1ffb640 1 -- 172.21.15.18:0/2572049264 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3338865003 0 0) 0x7f9cac133750 con 0x7f9cac143d10 2024-03-29T11:18:11.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.366+0000 7f9ca1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:11.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.366+0000 7f9ca1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:11.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.366+0000 7f9ca1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:11.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.367+0000 7f9ca1ffb640 10 monclient(hunting): my global_id is 4495 2024-03-29T11:18:11.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.367+0000 7f9ca1ffb640 1 -- 172.21.15.18:0/2572049264 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cac1420f0 con 0x7f9cac143d10 2024-03-29T11:18:11.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.367+0000 7f9ca1ffb640 1 -- 172.21.15.18:0/2572049264 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1257516200 0 0) 0x7f9cac1420f0 con 0x7f9cac143d10 2024-03-29T11:18:11.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.367+0000 7f9ca1ffb640 1 -- 172.21.15.18:0/2572049264 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9cac133750 con 0x7f9cac143d10 2024-03-29T11:18:11.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.367+0000 7f9ca1ffb640 1 -- 172.21.15.18:0/2572049264 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f9c9c001510 con 0x7f9cac143d10 2024-03-29T11:18:11.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.367+0000 7f9ca1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:11.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.367+0000 7f9ca1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:11.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.367+0000 7f9ca1ffb640 10 monclient(hunting): dump: 2024-03-29T11:18:11.352 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:18:11.352 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:11.352 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:11.352 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:11.352 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:11.352 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:11.352 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:18:11.353 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:18:11.353 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:18:11.353 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:11.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.367+0000 7f9ca1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:11.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.367+0000 7f9ca1ffb640 1 -- 172.21.15.18:0/2572049264 <== mon.1 v1:172.21.15.70:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4229471716 0 0) 0x7f9cac133750 con 0x7f9cac143d10 2024-03-29T11:18:11.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.367+0000 7f9ca1ffb640 1 -- 172.21.15.18:0/2572049264 >> v1:172.21.15.18:6790/0 conn(0x7f9cac0b1f60 legacy=0x7f9cac141090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:11.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.367+0000 7f9ca1ffb640 1 -- 172.21.15.18:0/2572049264 >> v1:172.21.15.18:6789/0 conn(0x7f9cac141a00 legacy=0x7f9cac133040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:11.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.367+0000 7f9ca1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:18:11.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.367+0000 7f9ca1ffb640 1 monclient: found mon.b 2024-03-29T11:18:11.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.367+0000 7f9ca1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:11.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.367+0000 7f9ca1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:18:11.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.367+0000 7f9ca1ffb640 1 -- 172.21.15.18:0/2572049264 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cac142c60 con 0x7f9cac143d10 2024-03-29T11:18:11.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.367+0000 7f9ca1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:18:11.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.367+0000 7f9ca1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:18:11.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.367+0000 7f9ca1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:11.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.367+0000 7f9cb3274640 5 monclient: authenticate success, global_id 4495 2024-03-29T11:18:11.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.367+0000 7f9cb3274640 10 monclient: _renew_subs 2024-03-29T11:18:11.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.367+0000 7f9cb3274640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:18:11.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.367+0000 7f9cb3274640 1 -- 172.21.15.18:0/2572049264 --> v1:172.21.15.70:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cac142f40 con 0x7f9cac143d10 2024-03-29T11:18:11.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.368+0000 7f9ca1ffb640 1 -- 172.21.15.18:0/2572049264 <== mon.1 v1:172.21.15.70:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c9c001a20 con 0x7f9cac143d10 2024-03-29T11:18:11.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.368+0000 7f9ca1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:11.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.368+0000 7f9ca1ffb640 1 -- 172.21.15.18:0/2572049264 <== mon.1 v1:172.21.15.70:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f9c9c003780 con 0x7f9cac143d10 2024-03-29T11:18:11.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.368+0000 7f9ca1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:11.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.368+0000 7f9ca1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:11.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.368+0000 7f9ca1ffb640 10 monclient: dump: 2024-03-29T11:18:11.354 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:18:11.354 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:11.354 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:11.355 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:11.355 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:11.355 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:11.355 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:18:11.355 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:18:11.355 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:18:11.355 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:11.355 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.368+0000 7f9cb3274640 10 client.4495.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:11.355 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.368+0000 7f9cb3274640 10 monclient: _renew_subs 2024-03-29T11:18:11.355 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.368+0000 7f9cb3274640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:18:11.355 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.368+0000 7f9cb3274640 1 -- 172.21.15.18:0/2572049264 --> v1:172.21.15.70:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cac138890 con 0x7f9cac143d10 2024-03-29T11:18:11.355 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.368+0000 7f9ca37fe640 4 set_mon_vals no callback set 2024-03-29T11:18:11.355 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.368+0000 7f9cb3274640 1 librados: init done 2024-03-29T11:18:11.355 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.369+0000 7f9cb3274640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:11.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.369+0000 7f9cb3274640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:11.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.369+0000 7f9cb3274640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:18:11.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.369+0000 7f9cb3274640 1 -- 172.21.15.18:0/2572049264 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cac075f10 con 0x7f9cac143d10 2024-03-29T11:18:11.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.370+0000 7f9ca1ffb640 1 -- 172.21.15.18:0/2572049264 <== mon.1 v1:172.21.15.70:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f9c9c019750 con 0x7f9cac143d10 2024-03-29T11:18:11.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.371+0000 7f9ca1ffb640 1 -- 172.21.15.18:0/2572049264 <== mon.1 v1:172.21.15.70:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f9c9c0836d0 con 0x7f9cac143d10 2024-03-29T11:18:11.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.371+0000 7f9ca1ffb640 10 client.4495.objecter ms_dispatch 0x7f9cac000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:11.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.371+0000 7f9ca1ffb640 3 client.4495.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:11.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.371+0000 7f9ca1ffb640 3 client.4495.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:11.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.371+0000 7f9ca1ffb640 20 client.4495.objecter dump_active .. 0 homeless 2024-03-29T11:18:11.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.371+0000 7f9cb1010640 10 monclient: get_auth_request con 0x7f9c9006dea0 auth_method 0 2024-03-29T11:18:11.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.372+0000 7f9ca1ffb640 10 client.4495.objecter ms_handle_connect 0x7f9c9006dea0 2024-03-29T11:18:11.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.379+0000 7f9ca1ffb640 1 -- 172.21.15.18:0/2572049264 <== mon.1 v1:172.21.15.70:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9c9c04efb0 con 0x7f9cac143d10 2024-03-29T11:18:11.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.379+0000 7f9ca1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:11.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.379+0000 7f9ca1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:11.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.693+0000 7f9cb3274640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:11.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.693+0000 7f9cb3274640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:11.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.693+0000 7f9cb3274640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:18:11.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.693+0000 7f9cb3274640 1 -- 172.21.15.18:0/2572049264 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9cac1417e0 con 0x7f9cac143d10 2024-03-29T11:18:11.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.694+0000 7f9ca1ffb640 1 -- 172.21.15.18:0/2572049264 <== mon.1 v1:172.21.15.70:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f9c9c050ee0 con 0x7f9cac143d10 2024-03-29T11:18:11.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.694+0000 7f9ca1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:11.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.694+0000 7f9ca1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:11.672 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:18:11.672 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:11.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.698+0000 7f9cb3274640 10 librados: watch_flush enter 2024-03-29T11:18:11.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.698+0000 7f9cb3274640 10 librados: watch_flush exit 2024-03-29T11:18:11.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.698+0000 7f9cb3274640 20 client.4495.objecter shutdown clearing up homeless session... 2024-03-29T11:18:11.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.698+0000 7f9cb3274640 10 client.4495.objecter successfully canceled tick 2024-03-29T11:18:11.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.698+0000 7f9cb3274640 1 -- 172.21.15.18:0/2572049264 >> v1:172.21.15.70:6816/3101871562 conn(0x7f9c9006dea0 legacy=0x7f9c90070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:11.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.698+0000 7f9cb3274640 10 monclient: shutdown 2024-03-29T11:18:11.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.698+0000 7f9cb3274640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:11.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.698+0000 7f9cb3274640 1 -- 172.21.15.18:0/2572049264 >> v1:172.21.15.70:6789/0 conn(0x7f9cac143d10 legacy=0x7f9cac132930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:11.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.699+0000 7f9cb3274640 1 -- 172.21.15.18:0/2572049264 shutdown_connections 2024-03-29T11:18:11.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.699+0000 7f9cb3274640 1 -- 172.21.15.18:0/2572049264 >> 172.21.15.18:0/2572049264 conn(0x7f9cac066a00 msgr2=0x7f9cac0b7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:11.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.700+0000 7f9cb3274640 1 -- 172.21.15.18:0/2572049264 shutdown_connections 2024-03-29T11:18:11.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.700+0000 7f9cb3274640 1 -- 172.21.15.18:0/2572049264 wait complete. 2024-03-29T11:18:11.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:11.700+0000 7f9cb3274640 1 librados: shutdown 2024-03-29T11:18:11.690 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:18.497 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:18.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.671+0000 7fd28be31640 10 monclient: get_monmap_and_config 2024-03-29T11:18:18.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.671+0000 7fd28be31640 10 monclient: build_initial_monmap 2024-03-29T11:18:18.649 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.671+0000 7fd28be31640 10 monclient: monmap: 2024-03-29T11:18:18.649 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:18:18.649 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:18.649 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:18:18.673128+0000 2024-03-29T11:18:18.649 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:18:18.673128+0000 2024-03-29T11:18:18.649 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:18.650 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:18.650 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:18:18.650 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:18:18.650 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:18:18.650 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:18.652 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.672+0000 7fd28be31640 1 Processor -- start 2024-03-29T11:18:18.652 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.672+0000 7fd28be31640 1 -- start start 2024-03-29T11:18:18.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.672+0000 7fd28be31640 10 monclient: init 2024-03-29T11:18:18.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.673+0000 7fd28be31640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:18.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.673+0000 7fd28be31640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:18:18.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.673+0000 7fd28be31640 10 monclient(hunting): picked mon.noname-a con 0x7fd2840b1f60 addr v1:172.21.15.18:6789/0 2024-03-29T11:18:18.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.673+0000 7fd28be31640 10 monclient(hunting): picked mon.noname-c con 0x7fd284071eb0 addr v1:172.21.15.70:6789/0 2024-03-29T11:18:18.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.673+0000 7fd28be31640 10 monclient(hunting): picked mon.noname-b con 0x7fd284143d10 addr v1:172.21.15.18:6790/0 2024-03-29T11:18:18.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.673+0000 7fd28be31640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd28406ef80 con 0x7fd2840b1f60 2024-03-29T11:18:18.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.673+0000 7fd28be31640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd28406f260 con 0x7fd284071eb0 2024-03-29T11:18:18.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.673+0000 7fd28be31640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2840b40c0 con 0x7fd284143d10 2024-03-29T11:18:18.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.673+0000 7fd28be31640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:18.653 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.673+0000 7fd28be31640 10 monclient(hunting): authenticate will time out at 1324.946700s 2024-03-29T11:18:18.654 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.676+0000 7fd2893cc640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7fd284071eb0 0x7fd284146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.18:48323/0 (socket says 172.21.15.18:48323) 2024-03-29T11:18:18.655 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.676+0000 7fd2893cc640 1 -- 172.21.15.18:0/764741853 learned_addr learned my addr 172.21.15.18:0/764741853 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:18:18.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.677+0000 7fd288bcb640 1 -- 172.21.15.18:0/764741853 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1264798406 0 0) 0x7fd2840b40c0 con 0x7fd284143d10 2024-03-29T11:18:18.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.677+0000 7fd288bcb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:18.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.677+0000 7fd288bcb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:18.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.677+0000 7fd288bcb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:18.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.677+0000 7fd288bcb640 10 monclient(hunting): my global_id is 4613 2024-03-29T11:18:18.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.677+0000 7fd288bcb640 1 -- 172.21.15.18:0/764741853 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd270003520 con 0x7fd284143d10 2024-03-29T11:18:18.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.678+0000 7fd288bcb640 1 -- 172.21.15.18:0/764741853 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4246134028 0 0) 0x7fd270003520 con 0x7fd284143d10 2024-03-29T11:18:18.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.678+0000 7fd288bcb640 1 -- 172.21.15.18:0/764741853 >> v1:172.21.15.70:6789/0 conn(0x7fd284071eb0 legacy=0x7fd284146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:18.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.678+0000 7fd288bcb640 1 -- 172.21.15.18:0/764741853 >> v1:172.21.15.18:6789/0 conn(0x7fd2840b1f60 legacy=0x7fd284143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:18:18.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.678+0000 7fd288bcb640 10 monclient: _finish_hunting 0 2024-03-29T11:18:18.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.678+0000 7fd288bcb640 1 monclient: found mon.noname-b 2024-03-29T11:18:18.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.678+0000 7fd288bcb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:18.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.678+0000 7fd288bcb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:18:18.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.678+0000 7fd288bcb640 1 -- 172.21.15.18:0/764741853 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd284155d70 con 0x7fd284143d10 2024-03-29T11:18:18.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.678+0000 7fd288bcb640 10 monclient: _finish_auth 0 2024-03-29T11:18:18.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.678+0000 7fd288bcb640 10 monclient: _check_auth_tickets 2024-03-29T11:18:18.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.678+0000 7fd288bcb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:18.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.678+0000 7fd288bcb640 1 -- 172.21.15.18:0/764741853 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fd27c001a10 con 0x7fd284143d10 2024-03-29T11:18:18.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.678+0000 7fd288bcb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:18.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.678+0000 7fd288bcb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:18.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.678+0000 7fd288bcb640 10 monclient: dump: 2024-03-29T11:18:18.660 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:18:18.660 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:18.660 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:18.660 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:18.660 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:18.660 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:18.661 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:18:18.661 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:18:18.661 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:18:18.661 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:18.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.678+0000 7fd288bcb640 1 -- 172.21.15.18:0/764741853 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd27c001ca0 con 0x7fd284143d10 2024-03-29T11:18:18.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.678+0000 7fd288bcb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:18.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.678+0000 7fd288bcb640 1 -- 172.21.15.18:0/764741853 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fd27c002030 con 0x7fd284143d10 2024-03-29T11:18:18.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.678+0000 7fd288bcb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:18.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.678+0000 7fd288bcb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:18.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.678+0000 7fd288bcb640 10 monclient: dump: 2024-03-29T11:18:18.661 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:18:18.661 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:18.661 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:18.661 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:18.661 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:18.662 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:18.662 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:18:18.662 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:18:18.662 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:18:18.662 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:18.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.679+0000 7fd28be31640 5 monclient: authenticate success, global_id 4613 2024-03-29T11:18:18.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.679+0000 7fd28be31640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:18.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.679+0000 7fd28be31640 4 set_mon_vals no callback set 2024-03-29T11:18:18.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.680+0000 7fd28be31640 10 monclient: shutdown 2024-03-29T11:18:18.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.680+0000 7fd28be31640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:18.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.680+0000 7fd28be31640 1 -- 172.21.15.18:0/764741853 >> v1:172.21.15.18:6790/0 conn(0x7fd284143d10 legacy=0x7fd284146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:18.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.680+0000 7fd28be31640 1 -- 172.21.15.18:0/764741853 shutdown_connections 2024-03-29T11:18:18.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.680+0000 7fd28be31640 1 -- 172.21.15.18:0/764741853 >> 172.21.15.18:0/764741853 conn(0x7fd284066a00 msgr2=0x7fd284066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:18.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.680+0000 7fd28be31640 1 -- 172.21.15.18:0/764741853 shutdown_connections 2024-03-29T11:18:18.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.680+0000 7fd28be31640 1 -- 172.21.15.18:0/764741853 wait complete. 2024-03-29T11:18:18.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.681+0000 7fd28be31640 10 monclient: build_initial_monmap 2024-03-29T11:18:18.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.681+0000 7fd28be31640 10 monclient: monmap: 2024-03-29T11:18:18.665 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:18:18.665 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:18.665 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:18:18.665 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:18:18.665 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:18.665 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:18.665 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:18:18.665 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:18:18.665 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:18:18.666 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:18.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.682+0000 7fd28be31640 1 librados: starting msgr at 2024-03-29T11:18:18.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.682+0000 7fd28be31640 1 librados: starting objecter 2024-03-29T11:18:18.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.682+0000 7fd28be31640 1 Processor -- start 2024-03-29T11:18:18.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.682+0000 7fd28be31640 1 -- start start 2024-03-29T11:18:18.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.682+0000 7fd28be31640 1 librados: setting wanted keys 2024-03-29T11:18:18.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.682+0000 7fd28be31640 1 librados: calling monclient init 2024-03-29T11:18:18.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.682+0000 7fd28be31640 10 monclient: init 2024-03-29T11:18:18.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.683+0000 7fd28be31640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:18.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.683+0000 7fd28be31640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:18:18.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.683+0000 7fd28be31640 10 monclient(hunting): picked mon.noname-c con 0x7fd2840b1f60 addr v1:172.21.15.18:6790/0 2024-03-29T11:18:18.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.685+0000 7fd28be31640 10 monclient(hunting): picked mon.noname-b con 0x7fd284143d10 addr v1:172.21.15.70:6789/0 2024-03-29T11:18:18.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.685+0000 7fd28be31640 10 monclient(hunting): picked mon.noname-a con 0x7fd284070980 addr v1:172.21.15.18:6789/0 2024-03-29T11:18:18.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.685+0000 7fd28be31640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd28414ad70 con 0x7fd284070980 2024-03-29T11:18:18.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.685+0000 7fd28be31640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd28413b860 con 0x7fd284143d10 2024-03-29T11:18:18.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.685+0000 7fd28be31640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd28414afe0 con 0x7fd2840b1f60 2024-03-29T11:18:18.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.685+0000 7fd28be31640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:18.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.685+0000 7fd28be31640 10 monclient(hunting): authenticate will time out at 1324.958058s 2024-03-29T11:18:18.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.686+0000 7fd289bcd640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fd2840b1f60 0x7fd284149c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:51211/0 (socket says 172.21.15.18:51211) 2024-03-29T11:18:18.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.686+0000 7fd289bcd640 1 -- 172.21.15.18:0/3493746631 learned_addr learned my addr 172.21.15.18:0/3493746631 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:18:18.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.688+0000 7fd27a7fc640 10 client.?.objecter ms_handle_connect 0x7fd2840b1f60 2024-03-29T11:18:18.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.688+0000 7fd27a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:18.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.688+0000 7fd27a7fc640 1 -- 172.21.15.18:0/3493746631 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1639243786 0 0) 0x7fd28414afe0 con 0x7fd2840b1f60 2024-03-29T11:18:18.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.688+0000 7fd27a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:18.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.688+0000 7fd27a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:18.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.688+0000 7fd27a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:18.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.688+0000 7fd27a7fc640 10 monclient(hunting): my global_id is 4616 2024-03-29T11:18:18.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.688+0000 7fd27a7fc640 1 -- 172.21.15.18:0/3493746631 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd268004eb0 con 0x7fd2840b1f60 2024-03-29T11:18:18.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.688+0000 7fd27a7fc640 10 client.?.objecter ms_handle_connect 0x7fd284143d10 2024-03-29T11:18:18.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.688+0000 7fd27a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:18.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.689+0000 7fd27a7fc640 1 -- 172.21.15.18:0/3493746631 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 163467201 0 0) 0x7fd28413b860 con 0x7fd284143d10 2024-03-29T11:18:18.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.689+0000 7fd27a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:18.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.689+0000 7fd27a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:18.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.689+0000 7fd27a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:18.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.689+0000 7fd27a7fc640 10 monclient(hunting): my global_id is 4498 2024-03-29T11:18:18.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.689+0000 7fd27a7fc640 1 -- 172.21.15.18:0/3493746631 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd28414afe0 con 0x7fd284143d10 2024-03-29T11:18:18.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.689+0000 7fd27a7fc640 1 -- 172.21.15.18:0/3493746631 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 684728753 0 0) 0x7fd268004eb0 con 0x7fd2840b1f60 2024-03-29T11:18:18.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.689+0000 7fd27a7fc640 1 -- 172.21.15.18:0/3493746631 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd28413b860 con 0x7fd2840b1f60 2024-03-29T11:18:18.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.689+0000 7fd27a7fc640 1 -- 172.21.15.18:0/3493746631 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fd2740018e0 con 0x7fd2840b1f60 2024-03-29T11:18:18.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.689+0000 7fd27a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:18.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.689+0000 7fd27a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:18.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.689+0000 7fd27a7fc640 10 monclient(hunting): dump: 2024-03-29T11:18:18.673 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:18:18.673 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:18.673 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:18.673 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:18.673 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:18.673 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:18.673 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:18:18.673 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:18:18.673 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:18:18.673 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:18.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.689+0000 7fd27a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:18.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.689+0000 7fd27a7fc640 1 -- 172.21.15.18:0/3493746631 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 372863726 0 0) 0x7fd28413b860 con 0x7fd2840b1f60 2024-03-29T11:18:18.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.689+0000 7fd27a7fc640 1 -- 172.21.15.18:0/3493746631 >> v1:172.21.15.70:6789/0 conn(0x7fd284143d10 legacy=0x7fd28413aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:18.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.689+0000 7fd27a7fc640 1 -- 172.21.15.18:0/3493746631 >> v1:172.21.15.18:6789/0 conn(0x7fd284070980 legacy=0x7fd28413b150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:18.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.689+0000 7fd27a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:18:18.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.689+0000 7fd27a7fc640 1 monclient: found mon.c 2024-03-29T11:18:18.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.689+0000 7fd27a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:18.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.689+0000 7fd27a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:18:18.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.689+0000 7fd27a7fc640 1 -- 172.21.15.18:0/3493746631 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd28414b2c0 con 0x7fd2840b1f60 2024-03-29T11:18:18.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.689+0000 7fd27a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:18:18.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.689+0000 7fd27a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:18:18.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.689+0000 7fd27a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:18.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.689+0000 7fd28be31640 5 monclient: authenticate success, global_id 4616 2024-03-29T11:18:18.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.689+0000 7fd28be31640 10 monclient: _renew_subs 2024-03-29T11:18:18.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.689+0000 7fd28be31640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:18:18.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.689+0000 7fd28be31640 1 -- 172.21.15.18:0/3493746631 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2841496e0 con 0x7fd2840b1f60 2024-03-29T11:18:18.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.689+0000 7fd28be31640 10 client.4616.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:18.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.689+0000 7fd28be31640 10 monclient: _renew_subs 2024-03-29T11:18:18.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.689+0000 7fd28be31640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:18:18.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.689+0000 7fd28be31640 1 -- 172.21.15.18:0/3493746631 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2841490b0 con 0x7fd2840b1f60 2024-03-29T11:18:18.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.689+0000 7fd28be31640 1 librados: init done 2024-03-29T11:18:18.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.690+0000 7fd27a7fc640 10 client.4616.objecter ms_handle_connect 0x7fd284070980 2024-03-29T11:18:18.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.690+0000 7fd27a7fc640 10 client.4616.objecter resend_mon_ops 2024-03-29T11:18:18.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.690+0000 7fd27a7fc640 1 -- 172.21.15.18:0/3493746631 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2740035f0 con 0x7fd2840b1f60 2024-03-29T11:18:18.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.690+0000 7fd27a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:18.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.690+0000 7fd27a7fc640 1 -- 172.21.15.18:0/3493746631 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fd274003a00 con 0x7fd2840b1f60 2024-03-29T11:18:18.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.690+0000 7fd27a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:18.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.690+0000 7fd27a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:18.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.690+0000 7fd27bfff640 4 set_mon_vals no callback set 2024-03-29T11:18:18.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.690+0000 7fd27a7fc640 10 monclient: dump: 2024-03-29T11:18:18.675 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:18:18.675 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:18.675 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:18.675 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:18.675 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:18.676 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:18.676 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:18:18.676 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:18:18.676 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:18:18.676 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:18.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.694+0000 7fd28be31640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:18.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.694+0000 7fd28be31640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:18.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.694+0000 7fd28be31640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:18:18.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.694+0000 7fd28be31640 1 -- 172.21.15.18:0/3493746631 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd26c005080 con 0x7fd2840b1f60 2024-03-29T11:18:18.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.695+0000 7fd27a7fc640 1 -- 172.21.15.18:0/3493746631 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7fd274003b60 con 0x7fd2840b1f60 2024-03-29T11:18:18.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.696+0000 7fd27a7fc640 1 -- 172.21.15.18:0/3493746631 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7fd274084a00 con 0x7fd2840b1f60 2024-03-29T11:18:18.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.696+0000 7fd27a7fc640 10 client.4616.objecter ms_dispatch 0x7fd284000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:18.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.696+0000 7fd27a7fc640 3 client.4616.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:18.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.696+0000 7fd27a7fc640 3 client.4616.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:18.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.696+0000 7fd27a7fc640 20 client.4616.objecter dump_active .. 0 homeless 2024-03-29T11:18:18.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.696+0000 7fd2893cc640 10 monclient: get_auth_request con 0x7fd26806dd10 auth_method 0 2024-03-29T11:18:18.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.696+0000 7fd27a7fc640 10 client.4616.objecter ms_handle_connect 0x7fd26806dd10 2024-03-29T11:18:18.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.705+0000 7fd27a7fc640 1 -- 172.21.15.18:0/3493746631 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd2740502e0 con 0x7fd2840b1f60 2024-03-29T11:18:18.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.705+0000 7fd27a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:18.683 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:18.705+0000 7fd27a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:19.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:19.029+0000 7fd28be31640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:19.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:19.029+0000 7fd28be31640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:19.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:19.029+0000 7fd28be31640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:18:19.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:19.029+0000 7fd28be31640 1 -- 172.21.15.18:0/3493746631 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd26c005330 con 0x7fd2840b1f60 2024-03-29T11:18:19.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:19.030+0000 7fd27a7fc640 1 -- 172.21.15.18:0/3493746631 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7fd274052210 con 0x7fd2840b1f60 2024-03-29T11:18:19.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:19.030+0000 7fd27a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:19.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:19.030+0000 7fd27a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:19.010 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:18:19.010 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:19.017 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:19.036+0000 7fd28be31640 10 librados: watch_flush enter 2024-03-29T11:18:19.017 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:19.038+0000 7fd28be31640 10 librados: watch_flush exit 2024-03-29T11:18:19.017 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:19.038+0000 7fd28be31640 20 client.4616.objecter shutdown clearing up homeless session... 2024-03-29T11:18:19.017 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:19.038+0000 7fd28be31640 10 client.4616.objecter successfully canceled tick 2024-03-29T11:18:19.017 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:19.038+0000 7fd28be31640 1 -- 172.21.15.18:0/3493746631 >> v1:172.21.15.70:6816/3101871562 conn(0x7fd26806dd10 legacy=0x7fd2680701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:19.017 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:19.038+0000 7fd28be31640 10 monclient: shutdown 2024-03-29T11:18:19.017 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:19.038+0000 7fd28be31640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:19.017 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:19.038+0000 7fd28be31640 1 -- 172.21.15.18:0/3493746631 >> v1:172.21.15.18:6790/0 conn(0x7fd2840b1f60 legacy=0x7fd284149c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:19.017 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:19.038+0000 7fd28be31640 1 -- 172.21.15.18:0/3493746631 shutdown_connections 2024-03-29T11:18:19.017 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:19.038+0000 7fd28be31640 1 -- 172.21.15.18:0/3493746631 >> 172.21.15.18:0/3493746631 conn(0x7fd284066a00 msgr2=0x7fd2840b7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:19.017 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:19.039+0000 7fd28be31640 1 -- 172.21.15.18:0/3493746631 shutdown_connections 2024-03-29T11:18:19.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:19.039+0000 7fd28be31640 1 -- 172.21.15.18:0/3493746631 wait complete. 2024-03-29T11:18:19.018 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:19.039+0000 7fd28be31640 1 librados: shutdown 2024-03-29T11:18:19.033 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:25.841 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:25.994 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.017+0000 7fe854b6c640 10 monclient: get_monmap_and_config 2024-03-29T11:18:25.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.017+0000 7fe854b6c640 10 monclient: build_initial_monmap 2024-03-29T11:18:25.995 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.017+0000 7fe854b6c640 10 monclient: monmap: 2024-03-29T11:18:25.995 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:18:25.995 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:25.995 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:18:26.018711+0000 2024-03-29T11:18:25.995 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:18:26.018711+0000 2024-03-29T11:18:25.995 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:25.995 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:25.995 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:18:25.995 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:18:25.995 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:18:25.995 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:25.996 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.018+0000 7fe854b6c640 1 Processor -- start 2024-03-29T11:18:25.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.018+0000 7fe854b6c640 1 -- start start 2024-03-29T11:18:25.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.018+0000 7fe854b6c640 10 monclient: init 2024-03-29T11:18:25.997 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.019+0000 7fe854b6c640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:25.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.019+0000 7fe854b6c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:18:25.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.019+0000 7fe854b6c640 10 monclient(hunting): picked mon.noname-b con 0x7fe8501457f0 addr v1:172.21.15.18:6790/0 2024-03-29T11:18:25.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.019+0000 7fe854b6c640 10 monclient(hunting): picked mon.noname-c con 0x7fe85013f650 addr v1:172.21.15.70:6789/0 2024-03-29T11:18:25.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.019+0000 7fe854b6c640 10 monclient(hunting): picked mon.noname-a con 0x7fe8500a9d70 addr v1:172.21.15.18:6789/0 2024-03-29T11:18:25.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.019+0000 7fe854b6c640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8500aaef0 con 0x7fe8500a9d70 2024-03-29T11:18:25.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.019+0000 7fe854b6c640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe850145bd0 con 0x7fe85013f650 2024-03-29T11:18:25.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.019+0000 7fe854b6c640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe850145eb0 con 0x7fe8501457f0 2024-03-29T11:18:25.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.019+0000 7fe854b6c640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:25.998 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.019+0000 7fe854b6c640 10 monclient(hunting): authenticate will time out at 1332.291794s 2024-03-29T11:18:25.999 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.022+0000 7fe84e59c640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fe8501457f0 0x7fe85013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:53720/0 (socket says 172.21.15.18:53720) 2024-03-29T11:18:25.999 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.022+0000 7fe84e59c640 1 -- 172.21.15.18:0/3432574229 learned_addr learned my addr 172.21.15.18:0/3432574229 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:18:26.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.024+0000 7fe84d59a640 1 -- 172.21.15.18:0/3432574229 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1346874585 0 0) 0x7fe850145eb0 con 0x7fe8501457f0 2024-03-29T11:18:26.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.024+0000 7fe84d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:18:26.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.024+0000 7fe84d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:26.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.024+0000 7fe84d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:26.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.024+0000 7fe84d59a640 10 monclient(hunting): my global_id is 4619 2024-03-29T11:18:26.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.024+0000 7fe84d59a640 1 -- 172.21.15.18:0/3432574229 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe838003520 con 0x7fe8501457f0 2024-03-29T11:18:26.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.024+0000 7fe84d59a640 1 -- 172.21.15.18:0/3432574229 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1675473838 0 0) 0x7fe838003520 con 0x7fe8501457f0 2024-03-29T11:18:26.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.024+0000 7fe84d59a640 1 -- 172.21.15.18:0/3432574229 >> v1:172.21.15.70:6789/0 conn(0x7fe85013f650 legacy=0x7fe8500b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:18:26.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.024+0000 7fe84d59a640 1 -- 172.21.15.18:0/3432574229 >> v1:172.21.15.18:6789/0 conn(0x7fe8500a9d70 legacy=0x7fe85014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:26.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.024+0000 7fe84d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:18:26.008 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.024+0000 7fe84d59a640 1 monclient: found mon.noname-b 2024-03-29T11:18:26.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.024+0000 7fe84d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:26.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.024+0000 7fe84d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:18:26.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.024+0000 7fe84d59a640 1 -- 172.21.15.18:0/3432574229 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe850146340 con 0x7fe8501457f0 2024-03-29T11:18:26.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.024+0000 7fe84d59a640 10 monclient: _finish_auth 0 2024-03-29T11:18:26.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.024+0000 7fe84d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:18:26.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.024+0000 7fe84d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:26.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.024+0000 7fe854b6c640 5 monclient: authenticate success, global_id 4619 2024-03-29T11:18:26.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.024+0000 7fe854b6c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:26.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.024+0000 7fe84d59a640 1 -- 172.21.15.18:0/3432574229 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fe840001690 con 0x7fe8501457f0 2024-03-29T11:18:26.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.024+0000 7fe84d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:26.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.024+0000 7fe84d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:26.009 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.024+0000 7fe84d59a640 10 monclient: dump: 2024-03-29T11:18:26.009 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:18:26.009 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:26.009 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:26.009 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:26.010 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:26.010 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:26.010 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:18:26.010 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:18:26.010 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:18:26.010 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:26.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.024+0000 7fe854b6c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:26.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.025+0000 7fe84d59a640 1 -- 172.21.15.18:0/3432574229 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe840001920 con 0x7fe8501457f0 2024-03-29T11:18:26.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.025+0000 7fe84d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:26.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.025+0000 7fe854b6c640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:26.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.025+0000 7fe854b6c640 4 set_mon_vals no callback set 2024-03-29T11:18:26.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.025+0000 7fe84d59a640 1 -- 172.21.15.18:0/3432574229 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fe840001cb0 con 0x7fe8501457f0 2024-03-29T11:18:26.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.025+0000 7fe84d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:26.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.025+0000 7fe84d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:26.010 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.025+0000 7fe84d59a640 10 monclient: dump: 2024-03-29T11:18:26.011 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:18:26.011 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:26.011 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:26.011 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:26.011 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:26.011 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:26.011 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:18:26.011 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:18:26.011 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:18:26.011 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:26.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.026+0000 7fe854b6c640 10 monclient: shutdown 2024-03-29T11:18:26.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.026+0000 7fe854b6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:26.011 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.026+0000 7fe854b6c640 1 -- 172.21.15.18:0/3432574229 >> v1:172.21.15.18:6790/0 conn(0x7fe8501457f0 legacy=0x7fe85013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:26.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.026+0000 7fe854b6c640 1 -- 172.21.15.18:0/3432574229 shutdown_connections 2024-03-29T11:18:26.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.026+0000 7fe854b6c640 1 -- 172.21.15.18:0/3432574229 >> 172.21.15.18:0/3432574229 conn(0x7fe850066a00 msgr2=0x7fe850066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:26.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.026+0000 7fe854b6c640 1 -- 172.21.15.18:0/3432574229 shutdown_connections 2024-03-29T11:18:26.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.026+0000 7fe854b6c640 1 -- 172.21.15.18:0/3432574229 wait complete. 2024-03-29T11:18:26.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.028+0000 7fe854b6c640 10 monclient: build_initial_monmap 2024-03-29T11:18:26.012 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.028+0000 7fe854b6c640 10 monclient: monmap: 2024-03-29T11:18:26.012 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:18:26.012 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:26.012 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:18:26.012 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:18:26.012 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:26.012 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:26.012 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:18:26.013 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:18:26.013 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:18:26.013 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:26.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.028+0000 7fe854b6c640 1 librados: starting msgr at 2024-03-29T11:18:26.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.028+0000 7fe854b6c640 1 librados: starting objecter 2024-03-29T11:18:26.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.028+0000 7fe854b6c640 1 Processor -- start 2024-03-29T11:18:26.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.028+0000 7fe854b6c640 1 -- start start 2024-03-29T11:18:26.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.028+0000 7fe854b6c640 1 librados: setting wanted keys 2024-03-29T11:18:26.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.028+0000 7fe854b6c640 1 librados: calling monclient init 2024-03-29T11:18:26.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.028+0000 7fe854b6c640 10 monclient: init 2024-03-29T11:18:26.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.029+0000 7fe854b6c640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:26.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.029+0000 7fe854b6c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:18:26.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.029+0000 7fe854b6c640 10 monclient(hunting): picked mon.noname-b con 0x7fe8500a9d70 addr v1:172.21.15.70:6789/0 2024-03-29T11:18:26.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.029+0000 7fe854b6c640 10 monclient(hunting): picked mon.noname-a con 0x7fe85013f650 addr v1:172.21.15.18:6789/0 2024-03-29T11:18:26.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.029+0000 7fe854b6c640 10 monclient(hunting): picked mon.noname-c con 0x7fe8501457f0 addr v1:172.21.15.18:6790/0 2024-03-29T11:18:26.013 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.029+0000 7fe854b6c640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe85006e8d0 con 0x7fe85013f650 2024-03-29T11:18:26.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.029+0000 7fe854b6c640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe85013a0a0 con 0x7fe8500a9d70 2024-03-29T11:18:26.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.029+0000 7fe854b6c640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8501e1770 con 0x7fe8501457f0 2024-03-29T11:18:26.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.029+0000 7fe854b6c640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:26.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.029+0000 7fe854b6c640 10 monclient(hunting): authenticate will time out at 1332.301867s 2024-03-29T11:18:26.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.030+0000 7fe84ed9d640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fe8501457f0 0x7fe8501e73e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:56970/0 (socket says 172.21.15.18:56970) 2024-03-29T11:18:26.014 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.030+0000 7fe84ed9d640 1 -- 172.21.15.18:0/1515821834 learned_addr learned my addr 172.21.15.18:0/1515821834 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:18:26.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.032+0000 7fe83effd640 10 client.?.objecter ms_handle_connect 0x7fe8501457f0 2024-03-29T11:18:26.023 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.032+0000 7fe83effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:26.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.032+0000 7fe83effd640 1 -- 172.21.15.18:0/1515821834 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1381845974 0 0) 0x7fe8501e1770 con 0x7fe8501457f0 2024-03-29T11:18:26.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.032+0000 7fe83effd640 10 monclient(hunting): _negotiate 2024-03-29T11:18:26.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.032+0000 7fe83effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:26.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.032+0000 7fe83effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:26.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.032+0000 7fe83effd640 10 monclient(hunting): my global_id is 4622 2024-03-29T11:18:26.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.032+0000 7fe83effd640 1 -- 172.21.15.18:0/1515821834 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe830004eb0 con 0x7fe8501457f0 2024-03-29T11:18:26.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.034+0000 7fe83effd640 10 client.?.objecter ms_handle_connect 0x7fe8500a9d70 2024-03-29T11:18:26.024 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.034+0000 7fe83effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:26.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.034+0000 7fe83effd640 1 -- 172.21.15.18:0/1515821834 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1566675061 0 0) 0x7fe85013a0a0 con 0x7fe8500a9d70 2024-03-29T11:18:26.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.034+0000 7fe83effd640 10 monclient(hunting): _negotiate 2024-03-29T11:18:26.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.034+0000 7fe83effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:26.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.034+0000 7fe83effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:26.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.034+0000 7fe83effd640 10 monclient(hunting): my global_id is 4501 2024-03-29T11:18:26.025 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.034+0000 7fe83effd640 1 -- 172.21.15.18:0/1515821834 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8501e1770 con 0x7fe8500a9d70 2024-03-29T11:18:26.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.034+0000 7fe83effd640 1 -- 172.21.15.18:0/1515821834 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1387827776 0 0) 0x7fe830004eb0 con 0x7fe8501457f0 2024-03-29T11:18:26.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.035+0000 7fe83effd640 1 -- 172.21.15.18:0/1515821834 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe85013a0a0 con 0x7fe8501457f0 2024-03-29T11:18:26.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.035+0000 7fe83effd640 1 -- 172.21.15.18:0/1515821834 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fe848001950 con 0x7fe8501457f0 2024-03-29T11:18:26.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.035+0000 7fe83effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:26.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.035+0000 7fe83effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:26.026 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.035+0000 7fe83effd640 10 monclient(hunting): dump: 2024-03-29T11:18:26.027 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:18:26.027 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:26.027 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:26.027 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:26.027 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:26.027 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:26.028 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:18:26.028 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:18:26.028 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:18:26.028 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:26.028 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.035+0000 7fe83effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:26.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.035+0000 7fe83effd640 1 -- 172.21.15.18:0/1515821834 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2093348182 0 0) 0x7fe85013a0a0 con 0x7fe8501457f0 2024-03-29T11:18:26.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.035+0000 7fe83effd640 1 -- 172.21.15.18:0/1515821834 >> v1:172.21.15.70:6789/0 conn(0x7fe8500a9d70 legacy=0x7fe8501397d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:26.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.035+0000 7fe83effd640 1 -- 172.21.15.18:0/1515821834 >> v1:172.21.15.18:6789/0 conn(0x7fe85013f650 legacy=0x7fe8501e1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:26.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.035+0000 7fe83effd640 10 monclient: _finish_hunting 0 2024-03-29T11:18:26.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.035+0000 7fe83effd640 1 monclient: found mon.c 2024-03-29T11:18:26.029 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.035+0000 7fe83effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:26.030 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.035+0000 7fe83effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:18:26.030 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.035+0000 7fe83effd640 1 -- 172.21.15.18:0/1515821834 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8501e1900 con 0x7fe8501457f0 2024-03-29T11:18:26.030 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.035+0000 7fe83effd640 10 monclient: _finish_auth 0 2024-03-29T11:18:26.030 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.035+0000 7fe83effd640 10 monclient: _check_auth_tickets 2024-03-29T11:18:26.030 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.035+0000 7fe83effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:26.031 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.035+0000 7fe854b6c640 5 monclient: authenticate success, global_id 4622 2024-03-29T11:18:26.031 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.035+0000 7fe854b6c640 10 monclient: _renew_subs 2024-03-29T11:18:26.031 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.035+0000 7fe854b6c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:18:26.031 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.035+0000 7fe854b6c640 1 -- 172.21.15.18:0/1515821834 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe850141e10 con 0x7fe8501457f0 2024-03-29T11:18:26.031 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.035+0000 7fe854b6c640 10 client.4622.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:26.031 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.035+0000 7fe854b6c640 10 monclient: _renew_subs 2024-03-29T11:18:26.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.035+0000 7fe854b6c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:18:26.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.035+0000 7fe854b6c640 1 -- 172.21.15.18:0/1515821834 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8501e7cd0 con 0x7fe8501457f0 2024-03-29T11:18:26.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.035+0000 7fe854b6c640 1 librados: init done 2024-03-29T11:18:26.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.036+0000 7fe83effd640 1 -- 172.21.15.18:0/1515821834 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe848001b30 con 0x7fe8501457f0 2024-03-29T11:18:26.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.036+0000 7fe83effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:26.032 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.036+0000 7fe83effd640 1 -- 172.21.15.18:0/1515821834 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fe848003a60 con 0x7fe8501457f0 2024-03-29T11:18:26.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.036+0000 7fe83effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:26.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.036+0000 7fe83effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:26.033 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.036+0000 7fe83effd640 10 monclient: dump: 2024-03-29T11:18:26.033 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:18:26.033 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:26.033 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:26.034 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:26.034 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:26.034 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:26.034 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:18:26.034 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:18:26.034 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:18:26.035 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:26.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.036+0000 7fe84cd99640 4 set_mon_vals no callback set 2024-03-29T11:18:26.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.038+0000 7fe83effd640 1 -- 172.21.15.18:0/1515821834 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7fe848003bc0 con 0x7fe8501457f0 2024-03-29T11:18:26.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.039+0000 7fe83effd640 1 -- 172.21.15.18:0/1515821834 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7fe848084ae0 con 0x7fe8501457f0 2024-03-29T11:18:26.035 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.039+0000 7fe83effd640 10 client.4622.objecter ms_dispatch 0x7fe850000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:26.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.039+0000 7fe83effd640 3 client.4622.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:26.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.039+0000 7fe83effd640 3 client.4622.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:26.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.039+0000 7fe83effd640 20 client.4622.objecter dump_active .. 0 homeless 2024-03-29T11:18:26.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.039+0000 7fe854b6c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:26.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.039+0000 7fe854b6c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:26.036 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.039+0000 7fe854b6c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:18:26.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.039+0000 7fe854b6c640 1 -- 172.21.15.18:0/1515821834 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe834005080 con 0x7fe8501457f0 2024-03-29T11:18:26.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.041+0000 7fe84e59c640 10 monclient: get_auth_request con 0x7fe83006db40 auth_method 0 2024-03-29T11:18:26.037 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.042+0000 7fe83effd640 10 client.4622.objecter ms_handle_connect 0x7fe83006db40 2024-03-29T11:18:26.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.051+0000 7fe83effd640 1 -- 172.21.15.18:0/1515821834 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe8480503c0 con 0x7fe8501457f0 2024-03-29T11:18:26.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.051+0000 7fe83effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:26.038 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.051+0000 7fe83effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:26.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.374+0000 7fe854b6c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:26.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.374+0000 7fe854b6c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:26.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.374+0000 7fe854b6c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:18:26.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.374+0000 7fe854b6c640 1 -- 172.21.15.18:0/1515821834 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe834005330 con 0x7fe8501457f0 2024-03-29T11:18:26.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.375+0000 7fe83effd640 1 -- 172.21.15.18:0/1515821834 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7fe8480522f0 con 0x7fe8501457f0 2024-03-29T11:18:26.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.375+0000 7fe83effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:26.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.375+0000 7fe83effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:26.353 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:18:26.353 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:26.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.379+0000 7fe854b6c640 10 librados: watch_flush enter 2024-03-29T11:18:26.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.380+0000 7fe854b6c640 10 librados: watch_flush exit 2024-03-29T11:18:26.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.380+0000 7fe854b6c640 20 client.4622.objecter shutdown clearing up homeless session... 2024-03-29T11:18:26.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.380+0000 7fe854b6c640 10 client.4622.objecter successfully canceled tick 2024-03-29T11:18:26.358 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.381+0000 7fe854b6c640 1 -- 172.21.15.18:0/1515821834 >> v1:172.21.15.70:6816/3101871562 conn(0x7fe83006db40 legacy=0x7fe830070000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:26.358 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.381+0000 7fe854b6c640 10 monclient: shutdown 2024-03-29T11:18:26.358 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.381+0000 7fe854b6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:26.358 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.381+0000 7fe854b6c640 1 -- 172.21.15.18:0/1515821834 >> v1:172.21.15.18:6790/0 conn(0x7fe8501457f0 legacy=0x7fe8501e73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:26.358 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.381+0000 7fe854b6c640 1 -- 172.21.15.18:0/1515821834 shutdown_connections 2024-03-29T11:18:26.358 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.381+0000 7fe854b6c640 1 -- 172.21.15.18:0/1515821834 >> 172.21.15.18:0/1515821834 conn(0x7fe850066a00 msgr2=0x7fe85013cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:26.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.381+0000 7fe854b6c640 1 -- 172.21.15.18:0/1515821834 shutdown_connections 2024-03-29T11:18:26.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.381+0000 7fe854b6c640 1 -- 172.21.15.18:0/1515821834 wait complete. 2024-03-29T11:18:26.361 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:26.384+0000 7fe854b6c640 1 librados: shutdown 2024-03-29T11:18:26.381 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:33.188 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:33.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.381+0000 7f401892c640 10 monclient: get_monmap_and_config 2024-03-29T11:18:33.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.381+0000 7f401892c640 10 monclient: build_initial_monmap 2024-03-29T11:18:33.360 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.381+0000 7f401892c640 10 monclient: monmap: 2024-03-29T11:18:33.360 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:18:33.360 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:33.360 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:18:33.382540+0000 2024-03-29T11:18:33.360 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:18:33.382540+0000 2024-03-29T11:18:33.360 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:33.360 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:33.360 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:18:33.360 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:18:33.360 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:18:33.361 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:33.361 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.382+0000 7f401892c640 1 Processor -- start 2024-03-29T11:18:33.361 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.383+0000 7f401892c640 1 -- start start 2024-03-29T11:18:33.361 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.383+0000 7f401892c640 10 monclient: init 2024-03-29T11:18:33.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.384+0000 7f401892c640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:33.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.384+0000 7f401892c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:18:33.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.384+0000 7f401892c640 10 monclient(hunting): picked mon.noname-a con 0x7f400c006410 addr v1:172.21.15.18:6789/0 2024-03-29T11:18:33.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.384+0000 7f401892c640 10 monclient(hunting): picked mon.noname-b con 0x7f400c02a640 addr v1:172.21.15.18:6790/0 2024-03-29T11:18:33.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.384+0000 7f401892c640 10 monclient(hunting): picked mon.noname-c con 0x7f400c0327d0 addr v1:172.21.15.70:6789/0 2024-03-29T11:18:33.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.384+0000 7f401892c640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f400c02c820 con 0x7f400c006410 2024-03-29T11:18:33.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.384+0000 7f401892c640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f400c02a120 con 0x7f400c0327d0 2024-03-29T11:18:33.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.384+0000 7f401892c640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f400c0c75b0 con 0x7f400c02a640 2024-03-29T11:18:33.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.384+0000 7f401892c640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:33.362 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.384+0000 7f401892c640 10 monclient(hunting): authenticate will time out at 1339.656272s 2024-03-29T11:18:33.364 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.387+0000 7f4013fff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f400c0327d0 0x7f400c0d1550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.18:43103/0 (socket says 172.21.15.18:43103) 2024-03-29T11:18:33.364 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.387+0000 7f4013fff640 1 -- 172.21.15.18:0/643821262 learned_addr learned my addr 172.21.15.18:0/643821262 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:18:33.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.390+0000 7f40127fc640 1 -- 172.21.15.18:0/643821262 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3174560261 0 0) 0x7f400c02a120 con 0x7f400c0327d0 2024-03-29T11:18:33.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.390+0000 7f40127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:33.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.390+0000 7f40127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:33.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.390+0000 7f40127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:33.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.390+0000 7f40127fc640 10 monclient(hunting): my global_id is 4504 2024-03-29T11:18:33.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.391+0000 7f40127fc640 1 -- 172.21.15.18:0/643821262 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4004003520 con 0x7f400c0327d0 2024-03-29T11:18:33.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.391+0000 7f40127fc640 1 -- 172.21.15.18:0/643821262 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4234150379 0 0) 0x7f400c02c820 con 0x7f400c006410 2024-03-29T11:18:33.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.391+0000 7f40127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:33.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.391+0000 7f40127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:33.368 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.391+0000 7f40127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:33.369 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.391+0000 7f40127fc640 10 monclient(hunting): my global_id is 4536 2024-03-29T11:18:33.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.391+0000 7f40127fc640 1 -- 172.21.15.18:0/643821262 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f400c02a120 con 0x7f400c006410 2024-03-29T11:18:33.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.391+0000 7f40127fc640 1 -- 172.21.15.18:0/643821262 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 506922004 0 0) 0x7f400c0c75b0 con 0x7f400c02a640 2024-03-29T11:18:33.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.392+0000 7f40127fc640 10 monclient(hunting): _negotiate 2024-03-29T11:18:33.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.392+0000 7f40127fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:33.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.392+0000 7f40127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:33.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.392+0000 7f40127fc640 10 monclient(hunting): my global_id is 4625 2024-03-29T11:18:33.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.392+0000 7f40127fc640 1 -- 172.21.15.18:0/643821262 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f400c02c820 con 0x7f400c02a640 2024-03-29T11:18:33.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.392+0000 7f40127fc640 1 -- 172.21.15.18:0/643821262 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3279029143 0 0) 0x7f4004003520 con 0x7f400c0327d0 2024-03-29T11:18:33.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.392+0000 7f40127fc640 1 -- 172.21.15.18:0/643821262 >> v1:172.21.15.18:6790/0 conn(0x7f400c02a640 legacy=0x7f400c0320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:33.371 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.392+0000 7f40127fc640 1 -- 172.21.15.18:0/643821262 >> v1:172.21.15.18:6789/0 conn(0x7f400c006410 legacy=0x7f400c0c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:33.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.392+0000 7f40127fc640 10 monclient: _finish_hunting 0 2024-03-29T11:18:33.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.392+0000 7f40127fc640 1 monclient: found mon.noname-c 2024-03-29T11:18:33.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.392+0000 7f40127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:33.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.392+0000 7f40127fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.70:6789/0 2024-03-29T11:18:33.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.392+0000 7f40127fc640 1 -- 172.21.15.18:0/643821262 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f400c033110 con 0x7f400c0327d0 2024-03-29T11:18:33.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.392+0000 7f40127fc640 10 monclient: _finish_auth 0 2024-03-29T11:18:33.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.392+0000 7f40127fc640 10 monclient: _check_auth_tickets 2024-03-29T11:18:33.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.392+0000 7f40127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:33.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.392+0000 7f40127fc640 1 -- 172.21.15.18:0/643821262 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f401404f730 con 0x7f400c0327d0 2024-03-29T11:18:33.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.392+0000 7f40127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:33.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.392+0000 7f40127fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:18:33.372 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.392+0000 7f40127fc640 10 monclient: dump: 2024-03-29T11:18:33.372 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:18:33.372 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:33.372 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:33.373 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:33.373 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:33.373 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:33.373 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:18:33.373 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:18:33.373 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:18:33.373 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:33.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.392+0000 7f401892c640 5 monclient: authenticate success, global_id 4504 2024-03-29T11:18:33.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.392+0000 7f401892c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:33.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.393+0000 7f40127fc640 1 -- 172.21.15.18:0/643821262 <== mon.1 v1:172.21.15.70:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40140008d0 con 0x7f400c0327d0 2024-03-29T11:18:33.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.393+0000 7f40127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:33.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.393+0000 7f401892c640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:33.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.393+0000 7f401892c640 4 set_mon_vals no callback set 2024-03-29T11:18:33.373 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.394+0000 7f401892c640 10 monclient: shutdown 2024-03-29T11:18:33.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.394+0000 7f401892c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:33.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.394+0000 7f401892c640 1 -- 172.21.15.18:0/643821262 >> v1:172.21.15.70:6789/0 conn(0x7f400c0327d0 legacy=0x7f400c0d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:33.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.394+0000 7f401892c640 1 -- 172.21.15.18:0/643821262 shutdown_connections 2024-03-29T11:18:33.374 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.394+0000 7f401892c640 1 -- 172.21.15.18:0/643821262 >> 172.21.15.18:0/643821262 conn(0x7f400c0058f0 msgr2=0x7f400c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:33.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.395+0000 7f401892c640 1 -- 172.21.15.18:0/643821262 shutdown_connections 2024-03-29T11:18:33.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.395+0000 7f401892c640 1 -- 172.21.15.18:0/643821262 wait complete. 2024-03-29T11:18:33.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.396+0000 7f401892c640 10 monclient: build_initial_monmap 2024-03-29T11:18:33.376 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.396+0000 7f401892c640 10 monclient: monmap: 2024-03-29T11:18:33.376 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:18:33.376 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:33.376 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:18:33.376 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:18:33.376 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:33.376 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:33.376 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:18:33.376 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:18:33.376 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:18:33.377 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:33.377 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.396+0000 7f401892c640 1 librados: starting msgr at 2024-03-29T11:18:33.377 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.396+0000 7f401892c640 1 librados: starting objecter 2024-03-29T11:18:33.377 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.396+0000 7f401892c640 1 Processor -- start 2024-03-29T11:18:33.377 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.397+0000 7f401892c640 1 -- start start 2024-03-29T11:18:33.377 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.397+0000 7f401892c640 1 librados: setting wanted keys 2024-03-29T11:18:33.377 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.397+0000 7f401892c640 1 librados: calling monclient init 2024-03-29T11:18:33.377 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.397+0000 7f401892c640 10 monclient: init 2024-03-29T11:18:33.377 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.397+0000 7f401892c640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:33.377 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.397+0000 7f401892c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:18:33.377 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.398+0000 7f401892c640 10 monclient(hunting): picked mon.noname-a con 0x7f400c006410 addr v1:172.21.15.18:6789/0 2024-03-29T11:18:33.377 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.398+0000 7f401892c640 10 monclient(hunting): picked mon.noname-b con 0x7f400c02a640 addr v1:172.21.15.70:6789/0 2024-03-29T11:18:33.377 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.398+0000 7f401892c640 10 monclient(hunting): picked mon.noname-c con 0x7f400c0327d0 addr v1:172.21.15.18:6790/0 2024-03-29T11:18:33.377 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.398+0000 7f401892c640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f400c15e4c0 con 0x7f400c006410 2024-03-29T11:18:33.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.398+0000 7f401892c640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f400c0c2f90 con 0x7f400c02a640 2024-03-29T11:18:33.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.398+0000 7f401892c640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f400c0c3120 con 0x7f400c0327d0 2024-03-29T11:18:33.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.398+0000 7f401892c640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:33.378 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.398+0000 7f401892c640 10 monclient(hunting): authenticate will time out at 1339.670065s 2024-03-29T11:18:33.380 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.401+0000 7f4013fff640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f400c0327d0 0x7f400c0c01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:54038/0 (socket says 172.21.15.18:54038) 2024-03-29T11:18:33.380 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.401+0000 7f4013fff640 1 -- 172.21.15.18:0/3432349448 learned_addr learned my addr 172.21.15.18:0/3432349448 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:18:33.380 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.401+0000 7f3ffffff640 10 client.?.objecter ms_handle_connect 0x7f400c0327d0 2024-03-29T11:18:33.380 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.401+0000 7f3ffffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:33.380 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.402+0000 7f3ffffff640 10 client.?.objecter ms_handle_connect 0x7f400c02a640 2024-03-29T11:18:33.380 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.402+0000 7f3ffffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:33.380 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.402+0000 7f3ffffff640 10 client.?.objecter ms_handle_connect 0x7f400c006410 2024-03-29T11:18:33.380 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.402+0000 7f3ffffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:33.380 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.402+0000 7f3ffffff640 1 -- 172.21.15.18:0/3432349448 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1341660473 0 0) 0x7f400c0c2f90 con 0x7f400c02a640 2024-03-29T11:18:33.380 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.402+0000 7f3ffffff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:33.380 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.402+0000 7f3ffffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:33.380 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.402+0000 7f3ffffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:33.380 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.402+0000 7f3ffffff640 10 monclient(hunting): my global_id is 4507 2024-03-29T11:18:33.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.402+0000 7f3ffffff640 1 -- 172.21.15.18:0/3432349448 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ff8004eb0 con 0x7f400c02a640 2024-03-29T11:18:33.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.402+0000 7f3ffffff640 1 -- 172.21.15.18:0/3432349448 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2860930711 0 0) 0x7f400c15e4c0 con 0x7f400c006410 2024-03-29T11:18:33.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.402+0000 7f3ffffff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:33.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.402+0000 7f3ffffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:33.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.402+0000 7f3ffffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:33.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.402+0000 7f3ffffff640 10 monclient(hunting): my global_id is 4539 2024-03-29T11:18:33.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.402+0000 7f3ffffff640 1 -- 172.21.15.18:0/3432349448 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f400c0c2f90 con 0x7f400c006410 2024-03-29T11:18:33.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.403+0000 7f3ffffff640 1 -- 172.21.15.18:0/3432349448 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 300316868 0 0) 0x7f3ff8004eb0 con 0x7f400c02a640 2024-03-29T11:18:33.381 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.403+0000 7f3ffffff640 1 -- 172.21.15.18:0/3432349448 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f400c15e4c0 con 0x7f400c02a640 2024-03-29T11:18:33.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.403+0000 7f3ffffff640 1 -- 172.21.15.18:0/3432349448 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f4008001860 con 0x7f400c02a640 2024-03-29T11:18:33.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.403+0000 7f3ffffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:33.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.403+0000 7f3ffffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:33.386 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.403+0000 7f3ffffff640 10 monclient(hunting): dump: 2024-03-29T11:18:33.386 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:18:33.386 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:33.386 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:33.386 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:33.386 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:33.386 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:33.386 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:18:33.387 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:18:33.387 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:18:33.387 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:33.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.403+0000 7f3ffffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:33.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.403+0000 7f3ffffff640 1 -- 172.21.15.18:0/3432349448 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3131426703 0 0) 0x7f400c0c2f90 con 0x7f400c006410 2024-03-29T11:18:33.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.403+0000 7f3ffffff640 1 -- 172.21.15.18:0/3432349448 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ff8006980 con 0x7f400c006410 2024-03-29T11:18:33.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.403+0000 7f3ffffff640 1 -- 172.21.15.18:0/3432349448 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f4000001740 con 0x7f400c006410 2024-03-29T11:18:33.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.403+0000 7f3ffffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:33.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.403+0000 7f3ffffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:33.387 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.403+0000 7f3ffffff640 10 monclient(hunting): dump: 2024-03-29T11:18:33.387 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:18:33.387 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:33.387 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:33.387 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:33.388 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:33.388 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:33.388 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:18:33.388 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:18:33.388 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:18:33.388 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:33.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.403+0000 7f3ffffff640 1 -- 172.21.15.18:0/3432349448 <== mon.1 v1:172.21.15.70:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1579264701 0 0) 0x7f400c15e4c0 con 0x7f400c02a640 2024-03-29T11:18:33.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.404+0000 7f3ffffff640 1 -- 172.21.15.18:0/3432349448 >> v1:172.21.15.18:6790/0 conn(0x7f400c0327d0 legacy=0x7f400c0c01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:33.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.404+0000 7f3ffffff640 1 -- 172.21.15.18:0/3432349448 >> v1:172.21.15.18:6789/0 conn(0x7f400c006410 legacy=0x7f400c0c2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:33.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.404+0000 7f3ffffff640 10 monclient: _finish_hunting 0 2024-03-29T11:18:33.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.404+0000 7f3ffffff640 1 monclient: found mon.b 2024-03-29T11:18:33.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.404+0000 7f3ffffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:33.388 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.404+0000 7f3ffffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:18:33.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.404+0000 7f3ffffff640 1 -- 172.21.15.18:0/3432349448 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f400c0c1720 con 0x7f400c02a640 2024-03-29T11:18:33.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.404+0000 7f3ffffff640 10 monclient: _finish_auth 0 2024-03-29T11:18:33.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.404+0000 7f3ffffff640 10 monclient: _check_auth_tickets 2024-03-29T11:18:33.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.404+0000 7f3ffffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:33.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.404+0000 7f401892c640 5 monclient: authenticate success, global_id 4507 2024-03-29T11:18:33.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.404+0000 7f401892c640 10 monclient: _renew_subs 2024-03-29T11:18:33.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.404+0000 7f401892c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:18:33.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.404+0000 7f401892c640 1 -- 172.21.15.18:0/3432349448 --> v1:172.21.15.70:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f400c0c1910 con 0x7f400c02a640 2024-03-29T11:18:33.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.404+0000 7f401892c640 10 client.4507.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:33.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.404+0000 7f401892c640 10 monclient: _renew_subs 2024-03-29T11:18:33.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.404+0000 7f401892c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:18:33.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.404+0000 7f401892c640 1 -- 172.21.15.18:0/3432349448 --> v1:172.21.15.70:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f400c0d4c90 con 0x7f400c02a640 2024-03-29T11:18:33.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.404+0000 7f401892c640 1 librados: init done 2024-03-29T11:18:33.389 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.405+0000 7f3ffdffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:33.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.405+0000 7f3ffdffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:33.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.405+0000 7f3ffdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:18:33.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.405+0000 7f3ffdffb640 1 -- 172.21.15.18:0/3432349448 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f400c004cd0 con 0x7f400c02a640 2024-03-29T11:18:33.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.405+0000 7f3ffffff640 1 -- 172.21.15.18:0/3432349448 <== mon.1 v1:172.21.15.70:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4008003610 con 0x7f400c02a640 2024-03-29T11:18:33.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.405+0000 7f3ffffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:33.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.405+0000 7f3ffffff640 1 -- 172.21.15.18:0/3432349448 <== mon.1 v1:172.21.15.70:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f4008003920 con 0x7f400c02a640 2024-03-29T11:18:33.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.405+0000 7f3ffffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:33.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.405+0000 7f3ffffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:18:33.390 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.405+0000 7f3ffffff640 10 monclient: dump: 2024-03-29T11:18:33.390 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:18:33.390 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:33.390 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:33.390 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:33.391 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:33.391 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:33.391 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:18:33.391 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:18:33.391 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:18:33.391 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:33.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.406+0000 7f4011ffb640 4 set_mon_vals no callback set 2024-03-29T11:18:33.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.408+0000 7f3ffffff640 1 -- 172.21.15.18:0/3432349448 <== mon.1 v1:172.21.15.70:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f40080198f0 con 0x7f400c02a640 2024-03-29T11:18:33.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.408+0000 7f3ffffff640 1 -- 172.21.15.18:0/3432349448 <== mon.1 v1:172.21.15.70:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f4008053530 con 0x7f400c02a640 2024-03-29T11:18:33.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.408+0000 7f3ffffff640 10 client.4507.objecter ms_dispatch 0x7f4014000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:33.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.408+0000 7f3ffffff640 3 client.4507.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:33.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.408+0000 7f3ffffff640 3 client.4507.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:33.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.408+0000 7f3ffffff640 20 client.4507.objecter dump_active .. 0 homeless 2024-03-29T11:18:33.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.408+0000 7f40137fe640 10 monclient: get_auth_request con 0x7f3ff806e050 auth_method 0 2024-03-29T11:18:33.391 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.409+0000 7f3ffffff640 10 client.4507.objecter ms_handle_connect 0x7f3ff806e050 2024-03-29T11:18:33.392 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.412+0000 7f3ffffff640 1 -- 172.21.15.18:0/3432349448 <== mon.1 v1:172.21.15.70:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f400804f7d0 con 0x7f400c02a640 2024-03-29T11:18:33.392 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.412+0000 7f3ffffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:33.392 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.412+0000 7f3ffffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:33.760 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.783+0000 7f3ffdffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:33.760 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.783+0000 7f3ffdffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:33.760 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.783+0000 7f3ffdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.70:6789/0 2024-03-29T11:18:33.760 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.783+0000 7f3ffdffb640 1 -- 172.21.15.18:0/3432349448 --> v1:172.21.15.70:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f400c0386b0 con 0x7f400c02a640 2024-03-29T11:18:33.760 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.783+0000 7f3ffffff640 1 -- 172.21.15.18:0/3432349448 <== mon.1 v1:172.21.15.70:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f400804f7d0 con 0x7f400c02a640 2024-03-29T11:18:33.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.783+0000 7f3ffffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:33.761 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.783+0000 7f3ffffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:33.761 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:18:33.761 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:33.764 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.787+0000 7f3ffdffb640 10 librados: watch_flush enter 2024-03-29T11:18:33.764 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.787+0000 7f3ffdffb640 10 librados: watch_flush exit 2024-03-29T11:18:33.764 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.787+0000 7f3ffdffb640 20 client.4507.objecter shutdown clearing up homeless session... 2024-03-29T11:18:33.764 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.787+0000 7f3ffdffb640 10 client.4507.objecter successfully canceled tick 2024-03-29T11:18:33.764 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.787+0000 7f3ffdffb640 1 -- 172.21.15.18:0/3432349448 >> v1:172.21.15.70:6816/3101871562 conn(0x7f3ff806e050 legacy=0x7f3ff8070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:33.764 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.787+0000 7f3ffdffb640 10 monclient: shutdown 2024-03-29T11:18:33.765 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.787+0000 7f3ffdffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:33.765 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.787+0000 7f3ffdffb640 1 -- 172.21.15.18:0/3432349448 >> v1:172.21.15.70:6789/0 conn(0x7f400c02a640 legacy=0x7f400c15ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:33.765 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.787+0000 7f3ffdffb640 1 -- 172.21.15.18:0/3432349448 shutdown_connections 2024-03-29T11:18:33.765 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.787+0000 7f3ffdffb640 1 -- 172.21.15.18:0/3432349448 >> 172.21.15.18:0/3432349448 conn(0x7f400c0058f0 msgr2=0x7f400c0043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:33.765 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.788+0000 7f3ffdffb640 1 -- 172.21.15.18:0/3432349448 shutdown_connections 2024-03-29T11:18:33.765 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.788+0000 7f3ffdffb640 1 -- 172.21.15.18:0/3432349448 wait complete. 2024-03-29T11:18:33.766 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:33.789+0000 7f3ffdffb640 1 librados: shutdown 2024-03-29T11:18:33.782 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:40.590 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:40.730 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.752+0000 7f58836f4640 10 monclient: get_monmap_and_config 2024-03-29T11:18:40.730 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.752+0000 7f58836f4640 10 monclient: build_initial_monmap 2024-03-29T11:18:40.730 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.752+0000 7f58836f4640 10 monclient: monmap: 2024-03-29T11:18:40.730 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:18:40.730 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:40.730 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:18:40.753566+0000 2024-03-29T11:18:40.730 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:18:40.753566+0000 2024-03-29T11:18:40.731 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:40.731 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:40.731 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:18:40.731 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:18:40.731 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:18:40.731 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:40.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.753+0000 7f58836f4640 1 Processor -- start 2024-03-29T11:18:40.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.753+0000 7f58836f4640 1 -- start start 2024-03-29T11:18:40.731 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.753+0000 7f58836f4640 10 monclient: init 2024-03-29T11:18:40.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.754+0000 7f58836f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:40.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.754+0000 7f58836f4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:18:40.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.754+0000 7f58836f4640 10 monclient(hunting): picked mon.noname-b con 0x7f587c0ae250 addr v1:172.21.15.18:6790/0 2024-03-29T11:18:40.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.754+0000 7f58836f4640 10 monclient(hunting): picked mon.noname-a con 0x7f587c13f970 addr v1:172.21.15.18:6789/0 2024-03-29T11:18:40.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.754+0000 7f58836f4640 10 monclient(hunting): picked mon.noname-c con 0x7f587c142800 addr v1:172.21.15.70:6789/0 2024-03-29T11:18:40.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.754+0000 7f58836f4640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f587c0afe00 con 0x7f587c13f970 2024-03-29T11:18:40.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.754+0000 7f58836f4640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f587c142c10 con 0x7f587c142800 2024-03-29T11:18:40.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.754+0000 7f58836f4640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f587c0afb20 con 0x7f587c0ae250 2024-03-29T11:18:40.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.754+0000 7f58836f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:40.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.754+0000 7f58836f4640 10 monclient(hunting): authenticate will time out at 1347.026483s 2024-03-29T11:18:40.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.755+0000 7f5880c8f640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f587c13f970 0x7f587c1420f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:41936/0 (socket says 172.21.15.18:41936) 2024-03-29T11:18:40.733 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.755+0000 7f5880c8f640 1 -- 172.21.15.18:0/595765270 learned_addr learned my addr 172.21.15.18:0/595765270 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:18:40.735 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.756+0000 7f5873fff640 1 -- 172.21.15.18:0/595765270 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3190849634 0 0) 0x7f587c0afe00 con 0x7f587c13f970 2024-03-29T11:18:40.735 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.756+0000 7f5873fff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:40.735 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.756+0000 7f5873fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:40.735 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.756+0000 7f5873fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:40.735 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.756+0000 7f5873fff640 10 monclient(hunting): my global_id is 4542 2024-03-29T11:18:40.736 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.756+0000 7f5873fff640 1 -- 172.21.15.18:0/595765270 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5868003520 con 0x7f587c13f970 2024-03-29T11:18:40.736 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.756+0000 7f5873fff640 1 -- 172.21.15.18:0/595765270 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2092590106 0 0) 0x7f587c0afb20 con 0x7f587c0ae250 2024-03-29T11:18:40.736 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.756+0000 7f5873fff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:40.736 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.756+0000 7f5873fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:40.736 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.756+0000 7f5873fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:40.736 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.756+0000 7f5873fff640 10 monclient(hunting): my global_id is 4631 2024-03-29T11:18:40.736 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.756+0000 7f5873fff640 1 -- 172.21.15.18:0/595765270 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f587c0afe00 con 0x7f587c0ae250 2024-03-29T11:18:40.736 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.757+0000 7f5873fff640 1 -- 172.21.15.18:0/595765270 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 103521244 0 0) 0x7f587c0afe00 con 0x7f587c0ae250 2024-03-29T11:18:40.736 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.757+0000 7f5873fff640 1 -- 172.21.15.18:0/595765270 >> v1:172.21.15.70:6789/0 conn(0x7f587c142800 legacy=0x7f587c14da10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:18:40.736 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.758+0000 7f5873fff640 1 -- 172.21.15.18:0/595765270 >> v1:172.21.15.18:6789/0 conn(0x7f587c13f970 legacy=0x7f587c1420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:40.736 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.758+0000 7f5873fff640 10 monclient: _finish_hunting 0 2024-03-29T11:18:40.736 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.758+0000 7f5873fff640 1 monclient: found mon.noname-b 2024-03-29T11:18:40.736 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.758+0000 7f5873fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:40.737 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.758+0000 7f5873fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:18:40.737 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.758+0000 7f5873fff640 1 -- 172.21.15.18:0/595765270 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f587c0aab80 con 0x7f587c0ae250 2024-03-29T11:18:40.737 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.758+0000 7f5873fff640 10 monclient: _finish_auth 0 2024-03-29T11:18:40.737 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.758+0000 7f5873fff640 10 monclient: _check_auth_tickets 2024-03-29T11:18:40.737 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.758+0000 7f5873fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:40.737 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.758+0000 7f5873fff640 1 -- 172.21.15.18:0/595765270 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f5878001520 con 0x7f587c0ae250 2024-03-29T11:18:40.737 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.758+0000 7f5873fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:40.737 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.758+0000 7f5873fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:40.737 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.758+0000 7f5873fff640 10 monclient: dump: 2024-03-29T11:18:40.737 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:18:40.737 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:40.737 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:40.737 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:40.737 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:40.737 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:40.738 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:18:40.738 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:18:40.738 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:18:40.738 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:40.738 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.758+0000 7f58836f4640 5 monclient: authenticate success, global_id 4631 2024-03-29T11:18:40.738 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.758+0000 7f58836f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:40.738 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.758+0000 7f5873fff640 1 -- 172.21.15.18:0/595765270 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5878001680 con 0x7f587c0ae250 2024-03-29T11:18:40.738 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.758+0000 7f5873fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:40.738 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.758+0000 7f58836f4640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:40.738 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.758+0000 7f58836f4640 4 set_mon_vals no callback set 2024-03-29T11:18:40.738 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.758+0000 7f5873fff640 1 -- 172.21.15.18:0/595765270 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f5878001ad0 con 0x7f587c0ae250 2024-03-29T11:18:40.739 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.758+0000 7f5873fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:40.739 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.758+0000 7f5873fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:40.739 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.758+0000 7f5873fff640 10 monclient: dump: 2024-03-29T11:18:40.739 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:18:40.739 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:40.739 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:40.739 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:40.739 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:40.739 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:40.739 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:18:40.739 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:18:40.739 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:18:40.739 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:40.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.759+0000 7f58836f4640 10 monclient: shutdown 2024-03-29T11:18:40.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.759+0000 7f58836f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:40.740 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.759+0000 7f58836f4640 1 -- 172.21.15.18:0/595765270 >> v1:172.21.15.18:6790/0 conn(0x7f587c0ae250 legacy=0x7f587c13f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:40.741 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.760+0000 7f58836f4640 1 -- 172.21.15.18:0/595765270 shutdown_connections 2024-03-29T11:18:40.741 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.760+0000 7f58836f4640 1 -- 172.21.15.18:0/595765270 >> 172.21.15.18:0/595765270 conn(0x7f587c0b35e0 msgr2=0x7f587c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:40.741 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.760+0000 7f58836f4640 1 -- 172.21.15.18:0/595765270 shutdown_connections 2024-03-29T11:18:40.741 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.760+0000 7f58836f4640 1 -- 172.21.15.18:0/595765270 wait complete. 2024-03-29T11:18:40.741 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.761+0000 7f58836f4640 10 monclient: build_initial_monmap 2024-03-29T11:18:40.741 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.761+0000 7f58836f4640 10 monclient: monmap: 2024-03-29T11:18:40.741 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:18:40.741 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:40.741 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:18:40.741 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:18:40.741 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:40.741 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:40.741 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:18:40.741 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:18:40.742 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:18:40.742 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:40.742 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.762+0000 7f58836f4640 1 librados: starting msgr at 2024-03-29T11:18:40.742 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.762+0000 7f58836f4640 1 librados: starting objecter 2024-03-29T11:18:40.742 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.762+0000 7f58836f4640 1 Processor -- start 2024-03-29T11:18:40.746 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.762+0000 7f58836f4640 1 -- start start 2024-03-29T11:18:40.746 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.762+0000 7f58836f4640 1 librados: setting wanted keys 2024-03-29T11:18:40.747 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.762+0000 7f58836f4640 1 librados: calling monclient init 2024-03-29T11:18:40.747 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.762+0000 7f58836f4640 10 monclient: init 2024-03-29T11:18:40.747 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.764+0000 7f58836f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:40.747 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.764+0000 7f58836f4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:18:40.747 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.764+0000 7f58836f4640 10 monclient(hunting): picked mon.noname-b con 0x7f587c0ae250 addr v1:172.21.15.70:6789/0 2024-03-29T11:18:40.747 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.764+0000 7f58836f4640 10 monclient(hunting): picked mon.noname-a con 0x7f587c13f970 addr v1:172.21.15.18:6789/0 2024-03-29T11:18:40.747 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.764+0000 7f58836f4640 10 monclient(hunting): picked mon.noname-c con 0x7f587c142800 addr v1:172.21.15.18:6790/0 2024-03-29T11:18:40.747 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.764+0000 7f58836f4640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f587c144140 con 0x7f587c13f970 2024-03-29T11:18:40.747 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.764+0000 7f58836f4640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f587c1456a0 con 0x7f587c0ae250 2024-03-29T11:18:40.747 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.764+0000 7f58836f4640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f587c1dc3f0 con 0x7f587c142800 2024-03-29T11:18:40.747 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.764+0000 7f58836f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:40.747 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.764+0000 7f58836f4640 10 monclient(hunting): authenticate will time out at 1347.036537s 2024-03-29T11:18:40.747 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.765+0000 7f5881c91640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f587c142800 0x7f587c1dbce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:46248/0 (socket says 172.21.15.18:46248) 2024-03-29T11:18:40.748 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.765+0000 7f5881c91640 1 -- 172.21.15.18:0/2651944195 learned_addr learned my addr 172.21.15.18:0/2651944195 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:18:40.748 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.765+0000 7f5871ffb640 10 client.?.objecter ms_handle_connect 0x7f587c142800 2024-03-29T11:18:40.748 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.765+0000 7f5871ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:40.748 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.765+0000 7f5871ffb640 10 client.?.objecter ms_handle_connect 0x7f587c13f970 2024-03-29T11:18:40.748 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.765+0000 7f5871ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:40.748 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.765+0000 7f5871ffb640 1 -- 172.21.15.18:0/2651944195 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3061059111 0 0) 0x7f587c144140 con 0x7f587c13f970 2024-03-29T11:18:40.748 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.765+0000 7f5871ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:40.748 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.765+0000 7f5871ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:40.748 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.765+0000 7f5871ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:40.748 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.765+0000 7f5871ffb640 10 monclient(hunting): my global_id is 4545 2024-03-29T11:18:40.748 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.765+0000 7f5871ffb640 1 -- 172.21.15.18:0/2651944195 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5860004eb0 con 0x7f587c13f970 2024-03-29T11:18:40.748 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.766+0000 7f5871ffb640 1 -- 172.21.15.18:0/2651944195 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 709471705 0 0) 0x7f587c1dc3f0 con 0x7f587c142800 2024-03-29T11:18:40.748 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.766+0000 7f5871ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:40.748 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.766+0000 7f5871ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:40.748 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.766+0000 7f5871ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:40.749 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.766+0000 7f5871ffb640 10 monclient(hunting): my global_id is 4634 2024-03-29T11:18:40.749 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.766+0000 7f5871ffb640 1 -- 172.21.15.18:0/2651944195 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f587c144140 con 0x7f587c142800 2024-03-29T11:18:40.749 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.766+0000 7f5871ffb640 1 -- 172.21.15.18:0/2651944195 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 638415375 0 0) 0x7f5860004eb0 con 0x7f587c13f970 2024-03-29T11:18:40.749 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.766+0000 7f5871ffb640 1 -- 172.21.15.18:0/2651944195 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f587c1dc3f0 con 0x7f587c13f970 2024-03-29T11:18:40.749 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.766+0000 7f5871ffb640 1 -- 172.21.15.18:0/2651944195 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f586c001a70 con 0x7f587c13f970 2024-03-29T11:18:40.749 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.766+0000 7f5871ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:40.749 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.766+0000 7f5871ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:18:40.749 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.766+0000 7f5871ffb640 10 monclient(hunting): dump: 2024-03-29T11:18:40.749 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:18:40.749 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:40.749 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:40.749 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:40.749 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:40.749 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:40.750 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:18:40.750 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:18:40.750 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:18:40.750 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:40.750 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.766+0000 7f5871ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:40.750 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.767+0000 7f5871ffb640 1 -- 172.21.15.18:0/2651944195 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3985834705 0 0) 0x7f587c144140 con 0x7f587c142800 2024-03-29T11:18:40.750 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.767+0000 7f5871ffb640 1 -- 172.21.15.18:0/2651944195 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5860004eb0 con 0x7f587c142800 2024-03-29T11:18:40.750 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.767+0000 7f5871ffb640 1 -- 172.21.15.18:0/2651944195 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f5874001a90 con 0x7f587c142800 2024-03-29T11:18:40.750 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.767+0000 7f5871ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:40.750 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.767+0000 7f5871ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:40.750 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.767+0000 7f5871ffb640 10 monclient(hunting): dump: 2024-03-29T11:18:40.750 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:18:40.750 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:40.750 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:40.751 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:40.751 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:40.751 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:40.751 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:18:40.751 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:18:40.751 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:18:40.751 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:40.751 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.767+0000 7f5871ffb640 1 -- 172.21.15.18:0/2651944195 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2388595641 0 0) 0x7f5860004eb0 con 0x7f587c142800 2024-03-29T11:18:40.751 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.767+0000 7f5871ffb640 1 -- 172.21.15.18:0/2651944195 >> v1:172.21.15.70:6789/0 conn(0x7f587c0ae250 legacy=0x7f587c1db5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:40.751 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.767+0000 7f5871ffb640 1 -- 172.21.15.18:0/2651944195 >> v1:172.21.15.18:6789/0 conn(0x7f587c13f970 legacy=0x7f587c13b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:40.751 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.767+0000 7f5871ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:18:40.751 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.767+0000 7f5871ffb640 1 monclient: found mon.c 2024-03-29T11:18:40.751 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.767+0000 7f5871ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:40.751 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.767+0000 7f5871ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:18:40.751 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.767+0000 7f5871ffb640 1 -- 172.21.15.18:0/2651944195 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f587c145910 con 0x7f587c142800 2024-03-29T11:18:40.752 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.767+0000 7f5871ffb640 10 monclient: _finish_auth 0 2024-03-29T11:18:40.752 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.767+0000 7f5871ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:18:40.752 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.767+0000 7f5871ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:40.752 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.768+0000 7f5871ffb640 1 -- 172.21.15.18:0/2651944195 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5874001fa0 con 0x7f587c142800 2024-03-29T11:18:40.752 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.768+0000 7f5871ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:40.752 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.768+0000 7f5871ffb640 1 -- 172.21.15.18:0/2651944195 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f5874004d80 con 0x7f587c142800 2024-03-29T11:18:40.752 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.768+0000 7f5871ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:40.752 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.768+0000 7f5871ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:40.752 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.768+0000 7f5871ffb640 10 monclient: dump: 2024-03-29T11:18:40.752 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:18:40.752 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:40.752 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:40.752 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:40.752 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:40.752 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:40.753 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:18:40.753 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:18:40.753 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:18:40.753 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:40.753 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.768+0000 7f58737fe640 4 set_mon_vals no callback set 2024-03-29T11:18:40.753 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.768+0000 7f58836f4640 5 monclient: authenticate success, global_id 4634 2024-03-29T11:18:40.753 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.768+0000 7f58836f4640 10 monclient: _renew_subs 2024-03-29T11:18:40.753 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.768+0000 7f58836f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:18:40.753 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.768+0000 7f58836f4640 1 -- 172.21.15.18:0/2651944195 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f587c145bf0 con 0x7f587c142800 2024-03-29T11:18:40.753 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.769+0000 7f58836f4640 10 client.4634.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:40.753 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.769+0000 7f58836f4640 10 monclient: _renew_subs 2024-03-29T11:18:40.753 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.769+0000 7f58836f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:18:40.753 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.769+0000 7f58836f4640 1 -- 172.21.15.18:0/2651944195 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f587c11f200 con 0x7f587c142800 2024-03-29T11:18:40.753 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.769+0000 7f58836f4640 1 librados: init done 2024-03-29T11:18:40.754 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.771+0000 7f5871ffb640 1 -- 172.21.15.18:0/2651944195 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f587401ad50 con 0x7f587c142800 2024-03-29T11:18:40.754 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.772+0000 7f5871ffb640 1 -- 172.21.15.18:0/2651944195 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f5874084ca0 con 0x7f587c142800 2024-03-29T11:18:40.755 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.772+0000 7f5871ffb640 10 client.4634.objecter ms_dispatch 0x7f587c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:40.755 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.772+0000 7f5871ffb640 3 client.4634.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:40.755 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.772+0000 7f5871ffb640 3 client.4634.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:40.755 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.772+0000 7f5871ffb640 20 client.4634.objecter dump_active .. 0 homeless 2024-03-29T11:18:40.755 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.772+0000 7f58836f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:40.755 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.772+0000 7f58836f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:40.755 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.772+0000 7f58836f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:18:40.755 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.772+0000 7f58836f4640 1 -- 172.21.15.18:0/2651944195 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f587c075f10 con 0x7f587c142800 2024-03-29T11:18:40.755 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.772+0000 7f5881490640 10 monclient: get_auth_request con 0x7f586006de80 auth_method 0 2024-03-29T11:18:40.755 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.773+0000 7f5871ffb640 10 client.4634.objecter ms_handle_connect 0x7f586006de80 2024-03-29T11:18:40.756 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.779+0000 7f5871ffb640 1 -- 172.21.15.18:0/2651944195 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5874050580 con 0x7f587c142800 2024-03-29T11:18:40.756 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.779+0000 7f5871ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:40.756 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:40.779+0000 7f5871ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:41.077 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:41.100+0000 7f58836f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:41.077 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:41.100+0000 7f58836f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:41.077 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:41.100+0000 7f58836f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:18:41.077 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:41.100+0000 7f58836f4640 1 -- 172.21.15.18:0/2651944195 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f587c06ab70 con 0x7f587c142800 2024-03-29T11:18:41.077 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:41.101+0000 7f5871ffb640 1 -- 172.21.15.18:0/2651944195 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f58740524b0 con 0x7f587c142800 2024-03-29T11:18:41.077 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:41.101+0000 7f5871ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:41.078 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:41.101+0000 7f5871ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:41.078 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:18:41.078 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:41.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:41.105+0000 7f58836f4640 10 librados: watch_flush enter 2024-03-29T11:18:41.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:41.105+0000 7f58836f4640 10 librados: watch_flush exit 2024-03-29T11:18:41.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:41.105+0000 7f58836f4640 20 client.4634.objecter shutdown clearing up homeless session... 2024-03-29T11:18:41.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:41.105+0000 7f58836f4640 10 client.4634.objecter successfully canceled tick 2024-03-29T11:18:41.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:41.105+0000 7f58836f4640 1 -- 172.21.15.18:0/2651944195 >> v1:172.21.15.70:6816/3101871562 conn(0x7f586006de80 legacy=0x7f5860070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:41.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:41.105+0000 7f58836f4640 10 monclient: shutdown 2024-03-29T11:18:41.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:41.105+0000 7f58836f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:41.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:41.105+0000 7f58836f4640 1 -- 172.21.15.18:0/2651944195 >> v1:172.21.15.18:6790/0 conn(0x7f587c142800 legacy=0x7f587c1dbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:41.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:41.105+0000 7f58836f4640 1 -- 172.21.15.18:0/2651944195 shutdown_connections 2024-03-29T11:18:41.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:41.105+0000 7f58836f4640 1 -- 172.21.15.18:0/2651944195 >> 172.21.15.18:0/2651944195 conn(0x7f587c0b35e0 msgr2=0x7f587c13e2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:41.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:41.108+0000 7f58836f4640 1 -- 172.21.15.18:0/2651944195 shutdown_connections 2024-03-29T11:18:41.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:41.108+0000 7f58836f4640 1 -- 172.21.15.18:0/2651944195 wait complete. 2024-03-29T11:18:41.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:41.108+0000 7f58836f4640 1 librados: shutdown 2024-03-29T11:18:41.101 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:47.908 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:48.062 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.085+0000 7f39c76ae640 10 monclient: get_monmap_and_config 2024-03-29T11:18:48.062 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.085+0000 7f39c76ae640 10 monclient: build_initial_monmap 2024-03-29T11:18:48.062 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.085+0000 7f39c76ae640 10 monclient: monmap: 2024-03-29T11:18:48.062 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:18:48.062 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:48.063 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:18:48.086128+0000 2024-03-29T11:18:48.063 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:18:48.086128+0000 2024-03-29T11:18:48.063 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:48.063 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:48.063 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:18:48.063 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:18:48.063 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:18:48.063 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:48.064 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.086+0000 7f39c76ae640 1 Processor -- start 2024-03-29T11:18:48.064 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.086+0000 7f39c76ae640 1 -- start start 2024-03-29T11:18:48.064 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.086+0000 7f39c76ae640 10 monclient: init 2024-03-29T11:18:48.066 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.087+0000 7f39c76ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:48.066 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.087+0000 7f39c76ae640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:18:48.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.087+0000 7f39c76ae640 10 monclient(hunting): picked mon.noname-b con 0x7f39c01457f0 addr v1:172.21.15.18:6790/0 2024-03-29T11:18:48.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.087+0000 7f39c76ae640 10 monclient(hunting): picked mon.noname-a con 0x7f39c013f650 addr v1:172.21.15.18:6789/0 2024-03-29T11:18:48.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.087+0000 7f39c76ae640 10 monclient(hunting): picked mon.noname-c con 0x7f39c00a9d70 addr v1:172.21.15.70:6789/0 2024-03-29T11:18:48.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.087+0000 7f39c76ae640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39c00aaef0 con 0x7f39c013f650 2024-03-29T11:18:48.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.087+0000 7f39c76ae640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39c0145bd0 con 0x7f39c00a9d70 2024-03-29T11:18:48.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.087+0000 7f39c76ae640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39c0145eb0 con 0x7f39c01457f0 2024-03-29T11:18:48.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.087+0000 7f39c76ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:48.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.087+0000 7f39c76ae640 10 monclient(hunting): authenticate will time out at 1354.359838s 2024-03-29T11:18:48.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.088+0000 7f39c5c4b640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f39c00a9d70 0x7f39c014d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.18:50060/0 (socket says 172.21.15.18:50060) 2024-03-29T11:18:48.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.088+0000 7f39c5c4b640 1 -- 172.21.15.18:0/2237726326 learned_addr learned my addr 172.21.15.18:0/2237726326 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:18:48.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.088+0000 7f39b7fff640 1 -- 172.21.15.18:0/2237726326 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3670568259 0 0) 0x7f39c0145eb0 con 0x7f39c01457f0 2024-03-29T11:18:48.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.088+0000 7f39b7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:48.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.088+0000 7f39b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:48.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.088+0000 7f39b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:48.067 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.088+0000 7f39b7fff640 10 monclient(hunting): my global_id is 4637 2024-03-29T11:18:48.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.089+0000 7f39b7fff640 1 -- 172.21.15.18:0/2237726326 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39ac003520 con 0x7f39c01457f0 2024-03-29T11:18:48.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.089+0000 7f39b7fff640 1 -- 172.21.15.18:0/2237726326 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2368303098 0 0) 0x7f39c0145bd0 con 0x7f39c00a9d70 2024-03-29T11:18:48.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.089+0000 7f39b7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:48.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.089+0000 7f39b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:48.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.089+0000 7f39b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:48.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.089+0000 7f39b7fff640 10 monclient(hunting): my global_id is 4510 2024-03-29T11:18:48.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.089+0000 7f39b7fff640 1 -- 172.21.15.18:0/2237726326 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39c0145eb0 con 0x7f39c00a9d70 2024-03-29T11:18:48.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.089+0000 7f39b7fff640 1 -- 172.21.15.18:0/2237726326 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2448619620 0 0) 0x7f39c00aaef0 con 0x7f39c013f650 2024-03-29T11:18:48.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.089+0000 7f39b7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:18:48.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.089+0000 7f39b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:48.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.089+0000 7f39b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:48.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.089+0000 7f39b7fff640 10 monclient(hunting): my global_id is 4548 2024-03-29T11:18:48.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.089+0000 7f39b7fff640 1 -- 172.21.15.18:0/2237726326 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39c0145bd0 con 0x7f39c013f650 2024-03-29T11:18:48.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.089+0000 7f39b7fff640 1 -- 172.21.15.18:0/2237726326 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2375304192 0 0) 0x7f39c0145bd0 con 0x7f39c013f650 2024-03-29T11:18:48.068 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.089+0000 7f39b7fff640 1 -- 172.21.15.18:0/2237726326 >> v1:172.21.15.18:6790/0 conn(0x7f39c01457f0 legacy=0x7f39c013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:48.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.089+0000 7f39b7fff640 1 -- 172.21.15.18:0/2237726326 >> v1:172.21.15.70:6789/0 conn(0x7f39c00a9d70 legacy=0x7f39c014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:48.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.089+0000 7f39b7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:18:48.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.089+0000 7f39b7fff640 1 monclient: found mon.noname-a 2024-03-29T11:18:48.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.089+0000 7f39b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:48.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.089+0000 7f39b7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-29T11:18:48.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.089+0000 7f39b7fff640 1 -- 172.21.15.18:0/2237726326 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c0146340 con 0x7f39c013f650 2024-03-29T11:18:48.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.089+0000 7f39b7fff640 10 monclient: _finish_auth 0 2024-03-29T11:18:48.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.089+0000 7f39b7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:18:48.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.089+0000 7f39b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:48.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.089+0000 7f39b7fff640 1 -- 172.21.15.18:0/2237726326 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f39b00015b0 con 0x7f39c013f650 2024-03-29T11:18:48.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.089+0000 7f39b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:48.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.089+0000 7f39b7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:18:48.069 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.089+0000 7f39b7fff640 10 monclient: dump: 2024-03-29T11:18:48.070 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:18:48.070 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:48.070 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:48.070 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:48.070 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:48.070 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:48.070 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:18:48.070 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:18:48.070 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:18:48.070 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:48.070 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.089+0000 7f39c76ae640 5 monclient: authenticate success, global_id 4548 2024-03-29T11:18:48.070 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.089+0000 7f39c76ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:48.076 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.090+0000 7f39b7fff640 1 -- 172.21.15.18:0/2237726326 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39b0001790 con 0x7f39c013f650 2024-03-29T11:18:48.076 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.090+0000 7f39b7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:48.076 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.090+0000 7f39b7fff640 1 -- 172.21.15.18:0/2237726326 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f39b0001be0 con 0x7f39c013f650 2024-03-29T11:18:48.077 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.090+0000 7f39b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:48.077 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.090+0000 7f39b7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:48.077 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.090+0000 7f39b7fff640 10 monclient: dump: 2024-03-29T11:18:48.077 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:18:48.077 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:48.077 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:48.077 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:48.077 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:48.077 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:48.077 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:18:48.077 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:18:48.077 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:18:48.077 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:48.078 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.090+0000 7f39c76ae640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:48.078 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.090+0000 7f39c76ae640 4 set_mon_vals no callback set 2024-03-29T11:18:48.078 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.090+0000 7f39c76ae640 10 monclient: shutdown 2024-03-29T11:18:48.078 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.090+0000 7f39c76ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:48.078 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.090+0000 7f39c76ae640 1 -- 172.21.15.18:0/2237726326 >> v1:172.21.15.18:6789/0 conn(0x7f39c013f650 legacy=0x7f39c00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:48.078 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.091+0000 7f39c76ae640 1 -- 172.21.15.18:0/2237726326 shutdown_connections 2024-03-29T11:18:48.078 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.091+0000 7f39c76ae640 1 -- 172.21.15.18:0/2237726326 >> 172.21.15.18:0/2237726326 conn(0x7f39c0066a00 msgr2=0x7f39c0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:48.078 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.091+0000 7f39c76ae640 1 -- 172.21.15.18:0/2237726326 shutdown_connections 2024-03-29T11:18:48.078 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.091+0000 7f39c76ae640 1 -- 172.21.15.18:0/2237726326 wait complete. 2024-03-29T11:18:48.078 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.092+0000 7f39c76ae640 10 monclient: build_initial_monmap 2024-03-29T11:18:48.078 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.092+0000 7f39c76ae640 10 monclient: monmap: 2024-03-29T11:18:48.078 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:18:48.079 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:48.079 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:18:48.079 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:18:48.079 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:48.079 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:48.079 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:18:48.079 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:18:48.079 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:18:48.079 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:48.079 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.092+0000 7f39c76ae640 1 librados: starting msgr at 2024-03-29T11:18:48.079 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.092+0000 7f39c76ae640 1 librados: starting objecter 2024-03-29T11:18:48.079 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.092+0000 7f39c76ae640 1 Processor -- start 2024-03-29T11:18:48.079 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.093+0000 7f39c76ae640 1 -- start start 2024-03-29T11:18:48.080 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.093+0000 7f39c76ae640 1 librados: setting wanted keys 2024-03-29T11:18:48.080 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.093+0000 7f39c76ae640 1 librados: calling monclient init 2024-03-29T11:18:48.080 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.093+0000 7f39c76ae640 10 monclient: init 2024-03-29T11:18:48.080 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.093+0000 7f39c76ae640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:48.080 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.093+0000 7f39c76ae640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:18:48.080 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.093+0000 7f39c76ae640 10 monclient(hunting): picked mon.noname-b con 0x7f39c00a9d70 addr v1:172.21.15.70:6789/0 2024-03-29T11:18:48.080 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.093+0000 7f39c76ae640 10 monclient(hunting): picked mon.noname-c con 0x7f39c013f650 addr v1:172.21.15.18:6790/0 2024-03-29T11:18:48.080 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.094+0000 7f39c544a640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f39c00a9d70 0x7f39c00718c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.18:50570/0 (socket says 172.21.15.18:50570) 2024-03-29T11:18:48.080 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.094+0000 7f39c76ae640 10 monclient(hunting): picked mon.noname-a con 0x7f39c01457f0 addr v1:172.21.15.18:6789/0 2024-03-29T11:18:48.080 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.094+0000 7f39c544a640 1 -- 172.21.15.18:0/4263796012 learned_addr learned my addr 172.21.15.18:0/4263796012 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:18:48.080 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.094+0000 7f39c76ae640 1 -- 172.21.15.18:0/4263796012 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39c006fc30 con 0x7f39c01457f0 2024-03-29T11:18:48.080 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.094+0000 7f39b5ffb640 10 client.?.objecter ms_handle_connect 0x7f39c00a9d70 2024-03-29T11:18:48.080 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.094+0000 7f39b5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:48.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.094+0000 7f39c76ae640 1 -- 172.21.15.18:0/4263796012 --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39c0138740 con 0x7f39c00a9d70 2024-03-29T11:18:48.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.094+0000 7f39c76ae640 1 -- 172.21.15.18:0/4263796012 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39c0071fd0 con 0x7f39c013f650 2024-03-29T11:18:48.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.094+0000 7f39c76ae640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:48.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.094+0000 7f39c76ae640 10 monclient(hunting): authenticate will time out at 1354.366385s 2024-03-29T11:18:48.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.094+0000 7f39b5ffb640 10 client.?.objecter ms_handle_connect 0x7f39c013f650 2024-03-29T11:18:48.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.094+0000 7f39b5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:48.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.094+0000 7f39b5ffb640 10 client.?.objecter ms_handle_connect 0x7f39c01457f0 2024-03-29T11:18:48.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.094+0000 7f39b5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:48.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.095+0000 7f39b5ffb640 1 -- 172.21.15.18:0/4263796012 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3349782309 0 0) 0x7f39c0138740 con 0x7f39c00a9d70 2024-03-29T11:18:48.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.095+0000 7f39b5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:48.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.095+0000 7f39b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:48.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.095+0000 7f39b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:48.081 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.095+0000 7f39b5ffb640 10 monclient(hunting): my global_id is 4513 2024-03-29T11:18:48.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.095+0000 7f39b5ffb640 1 -- 172.21.15.18:0/4263796012 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39a4004eb0 con 0x7f39c00a9d70 2024-03-29T11:18:48.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.095+0000 7f39b5ffb640 1 -- 172.21.15.18:0/4263796012 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2596551210 0 0) 0x7f39c006fc30 con 0x7f39c01457f0 2024-03-29T11:18:48.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.095+0000 7f39b5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:48.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.095+0000 7f39b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:48.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.095+0000 7f39b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:48.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.095+0000 7f39b5ffb640 10 monclient(hunting): my global_id is 4551 2024-03-29T11:18:48.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.095+0000 7f39b5ffb640 1 -- 172.21.15.18:0/4263796012 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39c0138740 con 0x7f39c01457f0 2024-03-29T11:18:48.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.095+0000 7f39b5ffb640 1 -- 172.21.15.18:0/4263796012 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 392077768 0 0) 0x7f39c0071fd0 con 0x7f39c013f650 2024-03-29T11:18:48.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.095+0000 7f39b5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:18:48.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.095+0000 7f39b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:48.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.095+0000 7f39b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:48.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.095+0000 7f39b5ffb640 10 monclient(hunting): my global_id is 4640 2024-03-29T11:18:48.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.095+0000 7f39b5ffb640 1 -- 172.21.15.18:0/4263796012 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39c006fc30 con 0x7f39c013f650 2024-03-29T11:18:48.082 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.095+0000 7f39b5ffb640 1 -- 172.21.15.18:0/4263796012 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3558523506 0 0) 0x7f39a4004eb0 con 0x7f39c00a9d70 2024-03-29T11:18:48.083 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.095+0000 7f39b5ffb640 1 -- 172.21.15.18:0/4263796012 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39c0071fd0 con 0x7f39c00a9d70 2024-03-29T11:18:48.083 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.096+0000 7f39b5ffb640 1 -- 172.21.15.18:0/4263796012 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f39bc001910 con 0x7f39c00a9d70 2024-03-29T11:18:48.083 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.096+0000 7f39b5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:48.083 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.096+0000 7f39b5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:48.083 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.096+0000 7f39b5ffb640 10 monclient(hunting): dump: 2024-03-29T11:18:48.083 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:18:48.083 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:48.083 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:48.083 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:48.083 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:48.083 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:48.083 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:18:48.083 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:18:48.084 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:18:48.084 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:48.084 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.096+0000 7f39b5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:48.084 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.096+0000 7f39b5ffb640 1 -- 172.21.15.18:0/4263796012 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 225460733 0 0) 0x7f39c0138740 con 0x7f39c01457f0 2024-03-29T11:18:48.084 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.096+0000 7f39b5ffb640 1 -- 172.21.15.18:0/4263796012 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39a4004eb0 con 0x7f39c01457f0 2024-03-29T11:18:48.084 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.096+0000 7f39b5ffb640 1 -- 172.21.15.18:0/4263796012 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f39b8001cb0 con 0x7f39c01457f0 2024-03-29T11:18:48.084 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.096+0000 7f39b5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:48.084 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.096+0000 7f39b5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:48.084 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.096+0000 7f39b5ffb640 10 monclient(hunting): dump: 2024-03-29T11:18:48.084 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:18:48.084 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:48.084 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:48.084 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:48.084 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:48.085 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:48.085 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:18:48.085 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:18:48.085 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:18:48.085 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:48.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.096+0000 7f39b5ffb640 1 -- 172.21.15.18:0/4263796012 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 743105645 0 0) 0x7f39c006fc30 con 0x7f39c013f650 2024-03-29T11:18:48.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.096+0000 7f39b5ffb640 1 -- 172.21.15.18:0/4263796012 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39c0138740 con 0x7f39c013f650 2024-03-29T11:18:48.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.096+0000 7f39b5ffb640 1 -- 172.21.15.18:0/4263796012 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f39b0001b60 con 0x7f39c013f650 2024-03-29T11:18:48.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.096+0000 7f39b5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:48.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.096+0000 7f39b5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:48.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.096+0000 7f39b5ffb640 10 monclient(hunting): dump: 2024-03-29T11:18:48.085 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:18:48.085 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:48.086 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:48.086 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:48.086 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:48.086 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:48.086 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:18:48.086 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:18:48.086 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:18:48.086 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:48.086 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.096+0000 7f39b5ffb640 1 -- 172.21.15.18:0/4263796012 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3131543341 0 0) 0x7f39c0138740 con 0x7f39c013f650 2024-03-29T11:18:48.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.096+0000 7f39b5ffb640 1 -- 172.21.15.18:0/4263796012 >> v1:172.21.15.70:6789/0 conn(0x7f39c00a9d70 legacy=0x7f39c00718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:48.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.096+0000 7f39b5ffb640 1 -- 172.21.15.18:0/4263796012 >> v1:172.21.15.18:6789/0 conn(0x7f39c01457f0 legacy=0x7f39c006de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:48.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.097+0000 7f39b5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:18:48.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.097+0000 7f39b5ffb640 1 monclient: found mon.c 2024-03-29T11:18:48.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.097+0000 7f39b5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:48.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.097+0000 7f39b5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:18:48.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.097+0000 7f39b5ffb640 1 -- 172.21.15.18:0/4263796012 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c01dfae0 con 0x7f39c013f650 2024-03-29T11:18:48.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.097+0000 7f39b5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:18:48.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.097+0000 7f39b5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:18:48.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.097+0000 7f39b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:48.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.097+0000 7f39c76ae640 5 monclient: authenticate success, global_id 4640 2024-03-29T11:18:48.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.097+0000 7f39c76ae640 10 monclient: _renew_subs 2024-03-29T11:18:48.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.097+0000 7f39c76ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:18:48.087 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.097+0000 7f39c76ae640 1 -- 172.21.15.18:0/4263796012 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39c006e5a0 con 0x7f39c013f650 2024-03-29T11:18:48.088 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.097+0000 7f39c76ae640 10 client.4640.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:48.088 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.097+0000 7f39c76ae640 10 monclient: _renew_subs 2024-03-29T11:18:48.088 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.097+0000 7f39c76ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:18:48.088 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.097+0000 7f39c76ae640 1 -- 172.21.15.18:0/4263796012 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f39c0140290 con 0x7f39c013f650 2024-03-29T11:18:48.088 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.097+0000 7f39c76ae640 1 librados: init done 2024-03-29T11:18:48.088 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.097+0000 7f39b5ffb640 1 -- 172.21.15.18:0/4263796012 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39b00038a0 con 0x7f39c013f650 2024-03-29T11:18:48.088 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.097+0000 7f39b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:48.088 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.097+0000 7f39b5ffb640 1 -- 172.21.15.18:0/4263796012 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f39b0003bb0 con 0x7f39c013f650 2024-03-29T11:18:48.088 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.097+0000 7f39b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:48.088 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.097+0000 7f39b5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:48.089 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.097+0000 7f39b5ffb640 10 monclient: dump: 2024-03-29T11:18:48.089 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:18:48.089 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:48.089 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:48.089 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:48.089 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:48.089 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:48.089 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:18:48.089 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:18:48.089 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:18:48.089 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:48.089 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.097+0000 7f39b77fe640 4 set_mon_vals no callback set 2024-03-29T11:18:48.089 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.099+0000 7f39c76ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:48.089 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.099+0000 7f39b5ffb640 1 -- 172.21.15.18:0/4263796012 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f39b0019b80 con 0x7f39c013f650 2024-03-29T11:18:48.089 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.099+0000 7f39c76ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:48.090 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.099+0000 7f39c76ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:18:48.090 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.099+0000 7f39c76ae640 1 -- 172.21.15.18:0/4263796012 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39a0005080 con 0x7f39c013f650 2024-03-29T11:18:48.091 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.100+0000 7f39c544a640 10 monclient: get_auth_request con 0x7f39a406e0a0 auth_method 0 2024-03-29T11:18:48.091 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.100+0000 7f39b5ffb640 1 -- 172.21.15.18:0/4263796012 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f39b0083890 con 0x7f39c013f650 2024-03-29T11:18:48.091 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.100+0000 7f39b5ffb640 10 client.4640.objecter ms_dispatch 0x7f39c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:48.091 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.100+0000 7f39b5ffb640 3 client.4640.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:48.091 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.100+0000 7f39b5ffb640 3 client.4640.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:48.091 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.101+0000 7f39b5ffb640 20 client.4640.objecter dump_active .. 0 homeless 2024-03-29T11:18:48.091 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.101+0000 7f39b5ffb640 10 client.4640.objecter ms_handle_connect 0x7f39a406e0a0 2024-03-29T11:18:48.091 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.110+0000 7f39b5ffb640 1 -- 172.21.15.18:0/4263796012 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f39b004f1d0 con 0x7f39c013f650 2024-03-29T11:18:48.091 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.110+0000 7f39b5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:48.091 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.110+0000 7f39b5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:48.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.488+0000 7f39c76ae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:48.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.488+0000 7f39c76ae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:48.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.488+0000 7f39c76ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:18:48.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.488+0000 7f39c76ae640 1 -- 172.21.15.18:0/4263796012 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39a0005330 con 0x7f39c013f650 2024-03-29T11:18:48.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.489+0000 7f39b5ffb640 1 -- 172.21.15.18:0/4263796012 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f39b0082f40 con 0x7f39c013f650 2024-03-29T11:18:48.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.489+0000 7f39b5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:48.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.489+0000 7f39b5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:48.467 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:18:48.467 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:48.469 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.492+0000 7f39c76ae640 10 librados: watch_flush enter 2024-03-29T11:18:48.469 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.493+0000 7f39c76ae640 10 librados: watch_flush exit 2024-03-29T11:18:48.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.493+0000 7f39c76ae640 20 client.4640.objecter shutdown clearing up homeless session... 2024-03-29T11:18:48.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.493+0000 7f39c76ae640 10 client.4640.objecter successfully canceled tick 2024-03-29T11:18:48.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.493+0000 7f39c76ae640 1 -- 172.21.15.18:0/4263796012 >> v1:172.21.15.70:6816/3101871562 conn(0x7f39a406e0a0 legacy=0x7f39a4070560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:48.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.493+0000 7f39c76ae640 10 monclient: shutdown 2024-03-29T11:18:48.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.493+0000 7f39c76ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:48.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.493+0000 7f39c76ae640 1 -- 172.21.15.18:0/4263796012 >> v1:172.21.15.18:6790/0 conn(0x7f39c013f650 legacy=0x7f39c01df3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:48.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.493+0000 7f39c76ae640 1 -- 172.21.15.18:0/4263796012 shutdown_connections 2024-03-29T11:18:48.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.493+0000 7f39c76ae640 1 -- 172.21.15.18:0/4263796012 >> 172.21.15.18:0/4263796012 conn(0x7f39c0066a00 msgr2=0x7f39c00b4790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:48.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.493+0000 7f39c76ae640 1 -- 172.21.15.18:0/4263796012 shutdown_connections 2024-03-29T11:18:48.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.493+0000 7f39c76ae640 1 -- 172.21.15.18:0/4263796012 wait complete. 2024-03-29T11:18:48.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:48.493+0000 7f39c76ae640 1 librados: shutdown 2024-03-29T11:18:48.484 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:18:55.291 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:18:55.439 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.462+0000 7fa940d4d640 10 monclient: get_monmap_and_config 2024-03-29T11:18:55.439 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.462+0000 7fa940d4d640 10 monclient: build_initial_monmap 2024-03-29T11:18:55.439 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.462+0000 7fa940d4d640 10 monclient: monmap: 2024-03-29T11:18:55.439 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:18:55.439 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:55.440 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:18:55.463481+0000 2024-03-29T11:18:55.440 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:18:55.463481+0000 2024-03-29T11:18:55.440 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:55.440 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:55.440 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:18:55.440 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:18:55.440 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:18:55.440 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:55.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.463+0000 7fa940d4d640 1 Processor -- start 2024-03-29T11:18:55.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.464+0000 7fa940d4d640 1 -- start start 2024-03-29T11:18:55.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.464+0000 7fa940d4d640 10 monclient: init 2024-03-29T11:18:55.441 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.465+0000 7fa940d4d640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:55.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.465+0000 7fa940d4d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:18:55.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.465+0000 7fa940d4d640 10 monclient(hunting): picked mon.noname-b con 0x7fa93c0b4790 addr v1:172.21.15.18:6790/0 2024-03-29T11:18:55.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.465+0000 7fa940d4d640 10 monclient(hunting): picked mon.noname-c con 0x7fa93c0aea90 addr v1:172.21.15.70:6789/0 2024-03-29T11:18:55.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.465+0000 7fa940d4d640 10 monclient(hunting): picked mon.noname-a con 0x7fa93c0b39f0 addr v1:172.21.15.18:6789/0 2024-03-29T11:18:55.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.465+0000 7fa940d4d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa93c0b0c60 con 0x7fa93c0b39f0 2024-03-29T11:18:55.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.465+0000 7fa940d4d640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa93c0af020 con 0x7fa93c0aea90 2024-03-29T11:18:55.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.465+0000 7fa940d4d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa93c13fbd0 con 0x7fa93c0b4790 2024-03-29T11:18:55.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.465+0000 7fa940d4d640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:55.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.465+0000 7fa940d4d640 10 monclient(hunting): authenticate will time out at 1361.737204s 2024-03-29T11:18:55.442 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.465+0000 7fa93a59c640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fa93c0b4790 0x7fa93c0b1080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:43570/0 (socket says 172.21.15.18:43570) 2024-03-29T11:18:55.443 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.465+0000 7fa93a59c640 1 -- 172.21.15.18:0/1653337550 learned_addr learned my addr 172.21.15.18:0/1653337550 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:18:55.444 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.467+0000 7fa93959a640 1 -- 172.21.15.18:0/1653337550 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2555759962 0 0) 0x7fa93c13fbd0 con 0x7fa93c0b4790 2024-03-29T11:18:55.444 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.467+0000 7fa93959a640 10 monclient(hunting): _negotiate 2024-03-29T11:18:55.444 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.467+0000 7fa93959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:55.444 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.467+0000 7fa93959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:55.444 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.467+0000 7fa93959a640 10 monclient(hunting): my global_id is 4643 2024-03-29T11:18:55.444 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.467+0000 7fa93959a640 1 -- 172.21.15.18:0/1653337550 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa924003520 con 0x7fa93c0b4790 2024-03-29T11:18:55.444 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.467+0000 7fa93959a640 1 -- 172.21.15.18:0/1653337550 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1131071430 0 0) 0x7fa924003520 con 0x7fa93c0b4790 2024-03-29T11:18:55.444 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.467+0000 7fa93959a640 1 -- 172.21.15.18:0/1653337550 >> v1:172.21.15.70:6789/0 conn(0x7fa93c0aea90 legacy=0x7fa93c0af540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:18:55.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.469+0000 7fa93959a640 1 -- 172.21.15.18:0/1653337550 >> v1:172.21.15.18:6789/0 conn(0x7fa93c0b39f0 legacy=0x7fa93c13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:18:55.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.469+0000 7fa93959a640 10 monclient: _finish_hunting 0 2024-03-29T11:18:55.446 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.469+0000 7fa93959a640 1 monclient: found mon.noname-b 2024-03-29T11:18:55.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.469+0000 7fa93959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:55.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.469+0000 7fa93959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:18:55.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.469+0000 7fa93959a640 1 -- 172.21.15.18:0/1653337550 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa93c13fd60 con 0x7fa93c0b4790 2024-03-29T11:18:55.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.469+0000 7fa93959a640 10 monclient: _finish_auth 0 2024-03-29T11:18:55.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.469+0000 7fa93959a640 10 monclient: _check_auth_tickets 2024-03-29T11:18:55.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.469+0000 7fa93959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:55.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.469+0000 7fa93959a640 1 -- 172.21.15.18:0/1653337550 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fa930001690 con 0x7fa93c0b4790 2024-03-29T11:18:55.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.469+0000 7fa93959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:55.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.469+0000 7fa93959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:18:55.447 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.469+0000 7fa93959a640 10 monclient: dump: 2024-03-29T11:18:55.447 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:18:55.447 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:55.447 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:55.448 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:55.448 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:55.448 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:55.448 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:18:55.448 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:18:55.448 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:18:55.448 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:55.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.470+0000 7fa940d4d640 5 monclient: authenticate success, global_id 4643 2024-03-29T11:18:55.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.470+0000 7fa940d4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:18:55.448 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.471+0000 7fa93959a640 1 -- 172.21.15.18:0/1653337550 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa930001860 con 0x7fa93c0b4790 2024-03-29T11:18:55.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.471+0000 7fa93959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:55.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.471+0000 7fa93959a640 1 -- 172.21.15.18:0/1653337550 <== mon.2 v1:172.21.15.18:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fa930001cb0 con 0x7fa93c0b4790 2024-03-29T11:18:55.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.471+0000 7fa93959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:55.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.471+0000 7fa93959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:18:55.449 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.471+0000 7fa93959a640 10 monclient: dump: 2024-03-29T11:18:55.449 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:18:55.449 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:55.449 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:55.449 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:55.449 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:55.449 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:55.449 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:18:55.450 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:18:55.450 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:18:55.450 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:55.450 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.473+0000 7fa940d4d640 10 monclient: get_monmap_and_config success 2024-03-29T11:18:55.450 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.473+0000 7fa940d4d640 4 set_mon_vals no callback set 2024-03-29T11:18:55.451 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.474+0000 7fa940d4d640 10 monclient: shutdown 2024-03-29T11:18:55.451 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.474+0000 7fa940d4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:55.451 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.474+0000 7fa940d4d640 1 -- 172.21.15.18:0/1653337550 >> v1:172.21.15.18:6790/0 conn(0x7fa93c0b4790 legacy=0x7fa93c0b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:55.451 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.474+0000 7fa940d4d640 1 -- 172.21.15.18:0/1653337550 shutdown_connections 2024-03-29T11:18:55.451 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.474+0000 7fa940d4d640 1 -- 172.21.15.18:0/1653337550 >> 172.21.15.18:0/1653337550 conn(0x7fa93c066a00 msgr2=0x7fa93c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:55.452 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.475+0000 7fa940d4d640 1 -- 172.21.15.18:0/1653337550 shutdown_connections 2024-03-29T11:18:55.452 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.475+0000 7fa940d4d640 1 -- 172.21.15.18:0/1653337550 wait complete. 2024-03-29T11:18:55.453 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.476+0000 7fa940d4d640 10 monclient: build_initial_monmap 2024-03-29T11:18:55.453 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.477+0000 7fa940d4d640 10 monclient: monmap: 2024-03-29T11:18:55.453 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:18:55.453 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:18:55.454 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:18:55.454 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:18:55.454 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:18:55.454 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:55.454 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:18:55.454 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:18:55.454 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:18:55.454 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:55.455 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.477+0000 7fa940d4d640 1 librados: starting msgr at 2024-03-29T11:18:55.455 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.478+0000 7fa940d4d640 1 librados: starting objecter 2024-03-29T11:18:55.455 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.478+0000 7fa940d4d640 1 Processor -- start 2024-03-29T11:18:55.455 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.478+0000 7fa940d4d640 1 -- start start 2024-03-29T11:18:55.455 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.478+0000 7fa940d4d640 1 librados: setting wanted keys 2024-03-29T11:18:55.455 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.478+0000 7fa940d4d640 1 librados: calling monclient init 2024-03-29T11:18:55.455 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.478+0000 7fa940d4d640 10 monclient: init 2024-03-29T11:18:55.457 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.479+0000 7fa940d4d640 10 monclient: _reopen_session rank -1 2024-03-29T11:18:55.457 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.479+0000 7fa940d4d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:18:55.457 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.479+0000 7fa940d4d640 10 monclient(hunting): picked mon.noname-b con 0x7fa93c0b39f0 addr v1:172.21.15.70:6789/0 2024-03-29T11:18:55.457 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.479+0000 7fa940d4d640 10 monclient(hunting): picked mon.noname-c con 0x7fa93c0b4790 addr v1:172.21.15.18:6790/0 2024-03-29T11:18:55.457 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.479+0000 7fa940d4d640 10 monclient(hunting): picked mon.noname-a con 0x7fa93c0a9770 addr v1:172.21.15.18:6789/0 2024-03-29T11:18:55.457 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.479+0000 7fa940d4d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa93c1d80f0 con 0x7fa93c0a9770 2024-03-29T11:18:55.457 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.479+0000 7fa940d4d640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa93c0aa640 con 0x7fa93c0b39f0 2024-03-29T11:18:55.457 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.479+0000 7fa940d4d640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa93c1d4e30 con 0x7fa93c0b4790 2024-03-29T11:18:55.457 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.479+0000 7fa940d4d640 10 monclient(hunting): _renew_subs 2024-03-29T11:18:55.457 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.479+0000 7fa940d4d640 10 monclient(hunting): authenticate will time out at 1361.751473s 2024-03-29T11:18:55.459 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.482+0000 7fa939d9b640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7fa93c0b4790 0x7fa93c0a9f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:44400/0 (socket says 172.21.15.18:44400) 2024-03-29T11:18:55.459 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.482+0000 7fa939d9b640 1 -- 172.21.15.18:0/3551264113 learned_addr learned my addr 172.21.15.18:0/3551264113 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:18:55.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.484+0000 7fa92affd640 10 client.?.objecter ms_handle_connect 0x7fa93c0b4790 2024-03-29T11:18:55.461 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.484+0000 7fa92affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:55.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.484+0000 7fa92affd640 1 -- 172.21.15.18:0/3551264113 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 485033400 0 0) 0x7fa93c1d4e30 con 0x7fa93c0b4790 2024-03-29T11:18:55.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.485+0000 7fa92affd640 10 monclient(hunting): _negotiate 2024-03-29T11:18:55.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.485+0000 7fa92affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:55.464 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.485+0000 7fa92affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:55.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.485+0000 7fa92affd640 10 monclient(hunting): my global_id is 4646 2024-03-29T11:18:55.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.485+0000 7fa92affd640 1 -- 172.21.15.18:0/3551264113 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa91c004eb0 con 0x7fa93c0b4790 2024-03-29T11:18:55.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.485+0000 7fa92affd640 10 client.?.objecter ms_handle_connect 0x7fa93c0a9770 2024-03-29T11:18:55.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.485+0000 7fa92affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:55.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.485+0000 7fa92affd640 10 client.?.objecter ms_handle_connect 0x7fa93c0b39f0 2024-03-29T11:18:55.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.485+0000 7fa92affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:18:55.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.485+0000 7fa92affd640 1 -- 172.21.15.18:0/3551264113 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3145624309 0 0) 0x7fa93c1d80f0 con 0x7fa93c0a9770 2024-03-29T11:18:55.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.485+0000 7fa92affd640 10 monclient(hunting): _negotiate 2024-03-29T11:18:55.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.485+0000 7fa92affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:55.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.485+0000 7fa92affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:55.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.485+0000 7fa92affd640 10 monclient(hunting): my global_id is 4554 2024-03-29T11:18:55.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.485+0000 7fa92affd640 1 -- 172.21.15.18:0/3551264113 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa93c1d4e30 con 0x7fa93c0a9770 2024-03-29T11:18:55.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.485+0000 7fa92affd640 1 -- 172.21.15.18:0/3551264113 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3086353253 0 0) 0x7fa93c0aa640 con 0x7fa93c0b39f0 2024-03-29T11:18:55.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.485+0000 7fa92affd640 10 monclient(hunting): _negotiate 2024-03-29T11:18:55.465 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.485+0000 7fa92affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:18:55.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.485+0000 7fa92affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:18:55.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.485+0000 7fa92affd640 10 monclient(hunting): my global_id is 4516 2024-03-29T11:18:55.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.485+0000 7fa92affd640 1 -- 172.21.15.18:0/3551264113 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa93c1d80f0 con 0x7fa93c0b39f0 2024-03-29T11:18:55.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.486+0000 7fa92affd640 1 -- 172.21.15.18:0/3551264113 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3352135421 0 0) 0x7fa93c1d4e30 con 0x7fa93c0a9770 2024-03-29T11:18:55.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.486+0000 7fa92affd640 1 -- 172.21.15.18:0/3551264113 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa93c0aa640 con 0x7fa93c0a9770 2024-03-29T11:18:55.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.486+0000 7fa92affd640 1 -- 172.21.15.18:0/3551264113 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fa934001890 con 0x7fa93c0a9770 2024-03-29T11:18:55.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.486+0000 7fa92affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:55.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.486+0000 7fa92affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:18:55.466 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.486+0000 7fa92affd640 10 monclient(hunting): dump: 2024-03-29T11:18:55.466 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:18:55.466 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:55.466 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:55.466 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:55.466 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:55.466 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:55.467 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:18:55.467 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:18:55.467 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:18:55.467 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:55.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.486+0000 7fa92affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:18:55.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.486+0000 7fa92affd640 1 -- 172.21.15.18:0/3551264113 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3262999990 0 0) 0x7fa93c0aa640 con 0x7fa93c0a9770 2024-03-29T11:18:55.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.486+0000 7fa92affd640 1 -- 172.21.15.18:0/3551264113 >> v1:172.21.15.18:6790/0 conn(0x7fa93c0b4790 legacy=0x7fa93c0a9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:55.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.486+0000 7fa92affd640 1 -- 172.21.15.18:0/3551264113 >> v1:172.21.15.70:6789/0 conn(0x7fa93c0b39f0 legacy=0x7fa93c1d79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:55.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.486+0000 7fa92affd640 10 monclient: _finish_hunting 0 2024-03-29T11:18:55.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.486+0000 7fa92affd640 1 monclient: found mon.a 2024-03-29T11:18:55.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.486+0000 7fa92affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:18:55.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.486+0000 7fa92affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:18:55.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.486+0000 7fa92affd640 1 -- 172.21.15.18:0/3551264113 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa93c1e6250 con 0x7fa93c0a9770 2024-03-29T11:18:55.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.486+0000 7fa92affd640 10 monclient: _finish_auth 0 2024-03-29T11:18:55.467 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.486+0000 7fa92affd640 10 monclient: _check_auth_tickets 2024-03-29T11:18:55.468 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.486+0000 7fa92affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:18:55.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.486+0000 7fa940d4d640 5 monclient: authenticate success, global_id 4554 2024-03-29T11:18:55.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.486+0000 7fa940d4d640 10 monclient: _renew_subs 2024-03-29T11:18:55.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.486+0000 7fa940d4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:18:55.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.486+0000 7fa940d4d640 1 -- 172.21.15.18:0/3551264113 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa93c1e3220 con 0x7fa93c0a9770 2024-03-29T11:18:55.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.486+0000 7fa940d4d640 10 client.4554.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:18:55.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.486+0000 7fa940d4d640 10 monclient: _renew_subs 2024-03-29T11:18:55.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.486+0000 7fa940d4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:18:55.470 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.487+0000 7fa940d4d640 1 -- 172.21.15.18:0/3551264113 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa93c1e36f0 con 0x7fa93c0a9770 2024-03-29T11:18:55.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.487+0000 7fa940d4d640 1 librados: init done 2024-03-29T11:18:55.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.489+0000 7fa92affd640 1 -- 172.21.15.18:0/3551264113 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa934001d80 con 0x7fa93c0a9770 2024-03-29T11:18:55.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.489+0000 7fa92affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:18:55.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.489+0000 7fa92affd640 1 -- 172.21.15.18:0/3551264113 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7fa934003c10 con 0x7fa93c0a9770 2024-03-29T11:18:55.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.489+0000 7fa92affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:18:55.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.489+0000 7fa92affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:18:55.471 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.489+0000 7fa92affd640 10 monclient: dump: 2024-03-29T11:18:55.471 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:18:55.471 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:18:55.471 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:55.471 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:18:55.471 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:18:55.471 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:18:55.472 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:18:55.472 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:18:55.472 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:18:55.472 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:18:55.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.489+0000 7fa938d99640 4 set_mon_vals no callback set 2024-03-29T11:18:55.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.490+0000 7fa940d4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:55.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.490+0000 7fa940d4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:55.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.490+0000 7fa940d4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:18:55.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.490+0000 7fa940d4d640 1 -- 172.21.15.18:0/3551264113 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa920005080 con 0x7fa93c0a9770 2024-03-29T11:18:55.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.492+0000 7fa92affd640 1 -- 172.21.15.18:0/3551264113 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7fa934019be0 con 0x7fa93c0a9770 2024-03-29T11:18:55.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.493+0000 7fa92affd640 1 -- 172.21.15.18:0/3551264113 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7fa934083a80 con 0x7fa93c0a9770 2024-03-29T11:18:55.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.493+0000 7fa93a59c640 10 monclient: get_auth_request con 0x7fa91c06def0 auth_method 0 2024-03-29T11:18:55.472 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.493+0000 7fa92affd640 10 client.4554.objecter ms_dispatch 0x7fa93c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:18:55.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.493+0000 7fa92affd640 3 client.4554.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:18:55.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.493+0000 7fa92affd640 3 client.4554.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:18:55.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.493+0000 7fa92affd640 20 client.4554.objecter dump_active .. 0 homeless 2024-03-29T11:18:55.473 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.493+0000 7fa92affd640 10 client.4554.objecter ms_handle_connect 0x7fa91c06def0 2024-03-29T11:18:55.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.500+0000 7fa92affd640 1 -- 172.21.15.18:0/3551264113 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa93404f360 con 0x7fa93c0a9770 2024-03-29T11:18:55.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.500+0000 7fa92affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:18:55.477 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.500+0000 7fa92affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:18:55.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.863+0000 7fa940d4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:55.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.863+0000 7fa940d4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:55.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.863+0000 7fa940d4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:18:55.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.863+0000 7fa940d4d640 1 -- 172.21.15.18:0/3551264113 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa920005330 con 0x7fa93c0a9770 2024-03-29T11:18:55.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.864+0000 7fa92affd640 1 -- 172.21.15.18:0/3551264113 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7fa934051290 con 0x7fa93c0a9770 2024-03-29T11:18:55.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.864+0000 7fa92affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:18:55.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.864+0000 7fa92affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:18:55.841 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:18:55.842 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:18:55.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.867+0000 7fa940d4d640 10 librados: watch_flush enter 2024-03-29T11:18:55.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.867+0000 7fa940d4d640 10 librados: watch_flush exit 2024-03-29T11:18:55.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.867+0000 7fa940d4d640 20 client.4554.objecter shutdown clearing up homeless session... 2024-03-29T11:18:55.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.867+0000 7fa940d4d640 10 client.4554.objecter successfully canceled tick 2024-03-29T11:18:55.844 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.867+0000 7fa940d4d640 1 -- 172.21.15.18:0/3551264113 >> v1:172.21.15.70:6816/3101871562 conn(0x7fa91c06def0 legacy=0x7fa91c0703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:55.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.867+0000 7fa940d4d640 10 monclient: shutdown 2024-03-29T11:18:55.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.867+0000 7fa940d4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:18:55.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.867+0000 7fa940d4d640 1 -- 172.21.15.18:0/3551264113 >> v1:172.21.15.18:6789/0 conn(0x7fa93c0a9770 legacy=0x7fa93c1e1b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:18:55.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.867+0000 7fa940d4d640 1 -- 172.21.15.18:0/3551264113 shutdown_connections 2024-03-29T11:18:55.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.867+0000 7fa940d4d640 1 -- 172.21.15.18:0/3551264113 >> 172.21.15.18:0/3551264113 conn(0x7fa93c066a00 msgr2=0x7fa93c0b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:18:55.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.867+0000 7fa940d4d640 1 -- 172.21.15.18:0/3551264113 shutdown_connections 2024-03-29T11:18:55.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.868+0000 7fa940d4d640 1 -- 172.21.15.18:0/3551264113 wait complete. 2024-03-29T11:18:55.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:18:55.868+0000 7fa940d4d640 1 librados: shutdown 2024-03-29T11:18:55.860 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:02.668 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:02.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.839+0000 7f108084a640 10 monclient: get_monmap_and_config 2024-03-29T11:19:02.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.839+0000 7f108084a640 10 monclient: build_initial_monmap 2024-03-29T11:19:02.816 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.839+0000 7f108084a640 10 monclient: monmap: 2024-03-29T11:19:02.816 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:19:02.816 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:02.816 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:19:02.840033+0000 2024-03-29T11:19:02.816 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:19:02.840033+0000 2024-03-29T11:19:02.816 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:02.816 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:02.816 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:19:02.817 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:19:02.817 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:19:02.817 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:02.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.840+0000 7f108084a640 1 Processor -- start 2024-03-29T11:19:02.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.840+0000 7f108084a640 1 -- start start 2024-03-29T11:19:02.817 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.840+0000 7f108084a640 10 monclient: init 2024-03-29T11:19:02.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.841+0000 7f108084a640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:02.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.841+0000 7f108084a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:02.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.841+0000 7f108084a640 10 monclient(hunting): picked mon.noname-b con 0x7f107c0b0770 addr v1:172.21.15.18:6790/0 2024-03-29T11:19:02.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.841+0000 7f108084a640 10 monclient(hunting): picked mon.noname-c con 0x7f107c14a700 addr v1:172.21.15.70:6789/0 2024-03-29T11:19:02.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.841+0000 7f108084a640 10 monclient(hunting): picked mon.noname-a con 0x7f107c0ac5a0 addr v1:172.21.15.18:6789/0 2024-03-29T11:19:02.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.841+0000 7f108084a640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107c0aba70 con 0x7f107c0ac5a0 2024-03-29T11:19:02.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.841+0000 7f108084a640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107c0aecb0 con 0x7f107c14a700 2024-03-29T11:19:02.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.841+0000 7f108084a640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107c0af150 con 0x7f107c0b0770 2024-03-29T11:19:02.818 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.841+0000 7f108084a640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:02.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.841+0000 7f108084a640 10 monclient(hunting): authenticate will time out at 1369.113243s 2024-03-29T11:19:02.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.842+0000 7f107a59c640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f107c0b0770 0x7f107c149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:43650/0 (socket says 172.21.15.18:43650) 2024-03-29T11:19:02.819 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.842+0000 7f107a59c640 1 -- 172.21.15.18:0/1900610702 learned_addr learned my addr 172.21.15.18:0/1900610702 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:19:02.823 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.843+0000 7f107959a640 1 -- 172.21.15.18:0/1900610702 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3391488245 0 0) 0x7f107c0af150 con 0x7f107c0b0770 2024-03-29T11:19:02.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.843+0000 7f107959a640 10 monclient(hunting): _negotiate 2024-03-29T11:19:02.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.843+0000 7f107959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:02.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.843+0000 7f107959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:02.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.843+0000 7f107959a640 10 monclient(hunting): my global_id is 4649 2024-03-29T11:19:02.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.843+0000 7f107959a640 1 -- 172.21.15.18:0/1900610702 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1064003520 con 0x7f107c0b0770 2024-03-29T11:19:02.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.844+0000 7f107959a640 1 -- 172.21.15.18:0/1900610702 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 748378594 0 0) 0x7f1064003520 con 0x7f107c0b0770 2024-03-29T11:19:02.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.844+0000 7f107959a640 1 -- 172.21.15.18:0/1900610702 >> v1:172.21.15.70:6789/0 conn(0x7f107c14a700 legacy=0x7f107c0abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:19:02.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.844+0000 7f107959a640 1 -- 172.21.15.18:0/1900610702 >> v1:172.21.15.18:6789/0 conn(0x7f107c0ac5a0 legacy=0x7f107c151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:19:02.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.844+0000 7f107959a640 10 monclient: _finish_hunting 0 2024-03-29T11:19:02.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.844+0000 7f107959a640 1 monclient: found mon.noname-b 2024-03-29T11:19:02.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.844+0000 7f107959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:02.824 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.845+0000 7f107959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.18:6790/0 2024-03-29T11:19:02.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.845+0000 7f107959a640 1 -- 172.21.15.18:0/1900610702 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f107c0ac9d0 con 0x7f107c0b0770 2024-03-29T11:19:02.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.845+0000 7f107959a640 10 monclient: _finish_auth 0 2024-03-29T11:19:02.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.845+0000 7f107959a640 10 monclient: _check_auth_tickets 2024-03-29T11:19:02.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.845+0000 7f107959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:02.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.845+0000 7f108084a640 5 monclient: authenticate success, global_id 4649 2024-03-29T11:19:02.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.845+0000 7f108084a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:02.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.845+0000 7f107959a640 1 -- 172.21.15.18:0/1900610702 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f106c001690 con 0x7f107c0b0770 2024-03-29T11:19:02.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.845+0000 7f107959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:02.825 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.845+0000 7f107959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:19:02.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.845+0000 7f107959a640 10 monclient: dump: 2024-03-29T11:19:02.826 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:19:02.826 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:02.826 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:02.826 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:02.826 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:02.826 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:02.826 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:19:02.826 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:19:02.826 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:19:02.826 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:02.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.845+0000 7f108084a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:02.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.845+0000 7f107959a640 1 -- 172.21.15.18:0/1900610702 <== mon.2 v1:172.21.15.18:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f106c0019a0 con 0x7f107c0b0770 2024-03-29T11:19:02.826 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.845+0000 7f107959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:02.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.845+0000 7f108084a640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:02.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.845+0000 7f108084a640 4 set_mon_vals no callback set 2024-03-29T11:19:02.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.846+0000 7f108084a640 10 monclient: shutdown 2024-03-29T11:19:02.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.846+0000 7f108084a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:02.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.846+0000 7f108084a640 1 -- 172.21.15.18:0/1900610702 >> v1:172.21.15.18:6790/0 conn(0x7f107c0b0770 legacy=0x7f107c149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:02.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.847+0000 7f108084a640 1 -- 172.21.15.18:0/1900610702 shutdown_connections 2024-03-29T11:19:02.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.847+0000 7f108084a640 1 -- 172.21.15.18:0/1900610702 >> 172.21.15.18:0/1900610702 conn(0x7f107c06b660 msgr2=0x7f107c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:02.827 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.847+0000 7f108084a640 1 -- 172.21.15.18:0/1900610702 shutdown_connections 2024-03-29T11:19:02.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.847+0000 7f108084a640 1 -- 172.21.15.18:0/1900610702 wait complete. 2024-03-29T11:19:02.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.848+0000 7f108084a640 10 monclient: build_initial_monmap 2024-03-29T11:19:02.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.848+0000 7f108084a640 10 monclient: monmap: 2024-03-29T11:19:02.829 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:19:02.829 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:02.829 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:19:02.829 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:19:02.829 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:02.829 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:02.829 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:19:02.829 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:19:02.829 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:19:02.829 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:02.829 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.848+0000 7f108084a640 1 librados: starting msgr at 2024-03-29T11:19:02.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.848+0000 7f108084a640 1 librados: starting objecter 2024-03-29T11:19:02.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.849+0000 7f108084a640 1 Processor -- start 2024-03-29T11:19:02.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.849+0000 7f108084a640 1 -- start start 2024-03-29T11:19:02.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.849+0000 7f108084a640 1 librados: setting wanted keys 2024-03-29T11:19:02.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.849+0000 7f108084a640 1 librados: calling monclient init 2024-03-29T11:19:02.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.849+0000 7f108084a640 10 monclient: init 2024-03-29T11:19:02.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.850+0000 7f108084a640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:02.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.850+0000 7f108084a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:02.830 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.852+0000 7f108084a640 10 monclient(hunting): picked mon.noname-a con 0x7f107c0ac5a0 addr v1:172.21.15.18:6789/0 2024-03-29T11:19:02.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.852+0000 7f107a59c640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f107c0ac5a0 0x7f107c143540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:55853/0 (socket says 172.21.15.18:55853) 2024-03-29T11:19:02.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.852+0000 7f107a59c640 1 -- 172.21.15.18:0/2831256086 learned_addr learned my addr 172.21.15.18:0/2831256086 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:19:02.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.852+0000 7f106affd640 10 client.?.objecter ms_handle_connect 0x7f107c0ac5a0 2024-03-29T11:19:02.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.852+0000 7f106affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:02.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.852+0000 7f108084a640 10 monclient(hunting): picked mon.noname-c con 0x7f107c0b0770 addr v1:172.21.15.18:6790/0 2024-03-29T11:19:02.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.852+0000 7f108084a640 10 monclient(hunting): picked mon.noname-b con 0x7f107c14a700 addr v1:172.21.15.70:6789/0 2024-03-29T11:19:02.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.852+0000 7f108084a640 1 -- 172.21.15.18:0/2831256086 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107c144f40 con 0x7f107c0ac5a0 2024-03-29T11:19:02.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.852+0000 7f108084a640 1 -- 172.21.15.18:0/2831256086 --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107c06f690 con 0x7f107c14a700 2024-03-29T11:19:02.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.853+0000 7f108084a640 1 -- 172.21.15.18:0/2831256086 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f107c06f970 con 0x7f107c0b0770 2024-03-29T11:19:02.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.853+0000 7f108084a640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:02.836 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.853+0000 7f106affd640 10 client.?.objecter ms_handle_connect 0x7f107c0b0770 2024-03-29T11:19:02.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.853+0000 7f106affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:02.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.853+0000 7f108084a640 10 monclient(hunting): authenticate will time out at 1369.125180s 2024-03-29T11:19:02.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.853+0000 7f106affd640 1 -- 172.21.15.18:0/2831256086 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3021980483 0 0) 0x7f107c144f40 con 0x7f107c0ac5a0 2024-03-29T11:19:02.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.853+0000 7f106affd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:02.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.853+0000 7f106affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:02.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.853+0000 7f106affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:02.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.853+0000 7f106affd640 10 monclient(hunting): my global_id is 4557 2024-03-29T11:19:02.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.853+0000 7f106affd640 1 -- 172.21.15.18:0/2831256086 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f105c004eb0 con 0x7f107c0ac5a0 2024-03-29T11:19:02.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.853+0000 7f106affd640 1 -- 172.21.15.18:0/2831256086 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3126705383 0 0) 0x7f105c004eb0 con 0x7f107c0ac5a0 2024-03-29T11:19:02.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.853+0000 7f106affd640 1 -- 172.21.15.18:0/2831256086 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f107c144f40 con 0x7f107c0ac5a0 2024-03-29T11:19:02.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.854+0000 7f106affd640 1 -- 172.21.15.18:0/2831256086 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f106c004b60 con 0x7f107c0ac5a0 2024-03-29T11:19:02.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.854+0000 7f106affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:02.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.854+0000 7f106affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:02.837 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.854+0000 7f106affd640 10 monclient(hunting): dump: 2024-03-29T11:19:02.837 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:19:02.838 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:02.838 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:02.838 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:02.838 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:02.838 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:02.838 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:19:02.838 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:19:02.838 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:19:02.838 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:02.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.854+0000 7f106affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:02.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.854+0000 7f106affd640 1 -- 172.21.15.18:0/2831256086 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 306033915 0 0) 0x7f107c144f40 con 0x7f107c0ac5a0 2024-03-29T11:19:02.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.854+0000 7f106affd640 1 -- 172.21.15.18:0/2831256086 >> v1:172.21.15.18:6790/0 conn(0x7f107c0b0770 legacy=0x7f107c1df020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:02.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.854+0000 7f106affd640 1 -- 172.21.15.18:0/2831256086 >> v1:172.21.15.70:6789/0 conn(0x7f107c14a700 legacy=0x7f107c1e62f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:19:02.838 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.856+0000 7f106affd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:02.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.856+0000 7f106affd640 1 monclient: found mon.a 2024-03-29T11:19:02.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.856+0000 7f106affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:02.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.856+0000 7f106affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:19:02.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.856+0000 7f106affd640 1 -- 172.21.15.18:0/2831256086 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f107c1df730 con 0x7f107c0ac5a0 2024-03-29T11:19:02.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.856+0000 7f106affd640 10 monclient: _finish_auth 0 2024-03-29T11:19:02.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.856+0000 7f106affd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:02.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.856+0000 7f106affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:02.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.856+0000 7f108084a640 5 monclient: authenticate success, global_id 4557 2024-03-29T11:19:02.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.856+0000 7f108084a640 10 monclient: _renew_subs 2024-03-29T11:19:02.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.856+0000 7f108084a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:19:02.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.856+0000 7f108084a640 1 -- 172.21.15.18:0/2831256086 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f107c1df9e0 con 0x7f107c0ac5a0 2024-03-29T11:19:02.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.856+0000 7f108084a640 10 client.4557.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:02.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.856+0000 7f108084a640 10 monclient: _renew_subs 2024-03-29T11:19:02.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.856+0000 7f108084a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:19:02.839 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.856+0000 7f108084a640 1 -- 172.21.15.18:0/2831256086 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f107c06dbb0 con 0x7f107c0ac5a0 2024-03-29T11:19:02.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.856+0000 7f108084a640 1 librados: init done 2024-03-29T11:19:02.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.858+0000 7f106affd640 1 -- 172.21.15.18:0/2831256086 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f106c001b90 con 0x7f107c0ac5a0 2024-03-29T11:19:02.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.858+0000 7f106affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:02.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.858+0000 7f106affd640 1 -- 172.21.15.18:0/2831256086 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f106c005150 con 0x7f107c0ac5a0 2024-03-29T11:19:02.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.858+0000 7f106affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:02.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.858+0000 7f106affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:02.840 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.858+0000 7f106affd640 10 monclient: dump: 2024-03-29T11:19:02.840 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:19:02.840 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:02.840 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:02.840 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:02.840 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:02.840 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:02.840 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:19:02.841 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:19:02.841 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:19:02.841 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:02.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.858+0000 7f1078d99640 4 set_mon_vals no callback set 2024-03-29T11:19:02.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.859+0000 7f106affd640 1 -- 172.21.15.18:0/2831256086 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f106c005370 con 0x7f107c0ac5a0 2024-03-29T11:19:02.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.859+0000 7f108084a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:02.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.859+0000 7f108084a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:02.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.859+0000 7f108084a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:19:02.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.859+0000 7f108084a640 1 -- 172.21.15.18:0/2831256086 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1060005080 con 0x7f107c0ac5a0 2024-03-29T11:19:02.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.860+0000 7f106affd640 1 -- 172.21.15.18:0/2831256086 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2023850011 0 0) 0x7f106c084f60 con 0x7f107c0ac5a0 2024-03-29T11:19:02.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.860+0000 7f106affd640 10 client.4557.objecter ms_dispatch 0x7f107c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:19:02.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.860+0000 7f106affd640 3 client.4557.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:19:02.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.860+0000 7f106affd640 3 client.4557.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:19:02.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.860+0000 7f106affd640 20 client.4557.objecter dump_active .. 0 homeless 2024-03-29T11:19:02.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.861+0000 7f1079d9b640 10 monclient: get_auth_request con 0x7f105c06dc00 auth_method 0 2024-03-29T11:19:02.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.861+0000 7f106affd640 10 client.4557.objecter ms_handle_connect 0x7f105c06dc00 2024-03-29T11:19:02.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.868+0000 7f106affd640 1 -- 172.21.15.18:0/2831256086 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f106c050840 con 0x7f107c0ac5a0 2024-03-29T11:19:02.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.868+0000 7f106affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:02.845 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:02.868+0000 7f106affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:03.180 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:03.202+0000 7f108084a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:03.180 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:03.202+0000 7f108084a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:03.180 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:03.202+0000 7f108084a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:19:03.180 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:03.202+0000 7f108084a640 1 -- 172.21.15.18:0/2831256086 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1060005330 con 0x7f107c0ac5a0 2024-03-29T11:19:03.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:03.204+0000 7f106affd640 1 -- 172.21.15.18:0/2831256086 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 863191377) 0x7f106c052770 con 0x7f107c0ac5a0 2024-03-29T11:19:03.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:03.204+0000 7f106affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:03.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:03.204+0000 7f106affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:03.183 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:19:03.183 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":10,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:09:06.242819+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:09:06.242819+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:03.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:03.209+0000 7f108084a640 10 librados: watch_flush enter 2024-03-29T11:19:03.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:03.209+0000 7f108084a640 10 librados: watch_flush exit 2024-03-29T11:19:03.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:03.209+0000 7f108084a640 20 client.4557.objecter shutdown clearing up homeless session... 2024-03-29T11:19:03.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:03.209+0000 7f108084a640 10 client.4557.objecter successfully canceled tick 2024-03-29T11:19:03.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:03.209+0000 7f108084a640 1 -- 172.21.15.18:0/2831256086 >> v1:172.21.15.70:6816/3101871562 conn(0x7f105c06dc00 legacy=0x7f105c0700c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:03.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:03.209+0000 7f108084a640 10 monclient: shutdown 2024-03-29T11:19:03.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:03.209+0000 7f108084a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:03.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:03.209+0000 7f108084a640 1 -- 172.21.15.18:0/2831256086 >> v1:172.21.15.18:6789/0 conn(0x7f107c0ac5a0 legacy=0x7f107c143540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:03.187 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:03.209+0000 7f108084a640 1 -- 172.21.15.18:0/2831256086 shutdown_connections 2024-03-29T11:19:03.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:03.209+0000 7f108084a640 1 -- 172.21.15.18:0/2831256086 >> 172.21.15.18:0/2831256086 conn(0x7f107c06b660 msgr2=0x7f107c149110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:03.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:03.210+0000 7f108084a640 1 -- 172.21.15.18:0/2831256086 shutdown_connections 2024-03-29T11:19:03.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:03.210+0000 7f108084a640 1 -- 172.21.15.18:0/2831256086 wait complete. 2024-03-29T11:19:03.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:03.210+0000 7f108084a640 1 librados: shutdown 2024-03-29T11:19:03.204 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:10.011 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:10.161 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.184+0000 7f081914b640 10 monclient: get_monmap_and_config 2024-03-29T11:19:10.161 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.184+0000 7f081914b640 10 monclient: build_initial_monmap 2024-03-29T11:19:10.161 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.184+0000 7f081914b640 10 monclient: monmap: 2024-03-29T11:19:10.161 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:19:10.161 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:10.162 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:19:10.185083+0000 2024-03-29T11:19:10.162 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:19:10.185083+0000 2024-03-29T11:19:10.162 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:10.162 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:10.162 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:19:10.162 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:19:10.162 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:19:10.162 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:10.163 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.185+0000 7f081914b640 1 Processor -- start 2024-03-29T11:19:10.163 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.185+0000 7f081914b640 1 -- start start 2024-03-29T11:19:10.163 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.185+0000 7f081914b640 10 monclient: init 2024-03-29T11:19:10.163 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.186+0000 7f081914b640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:10.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.186+0000 7f081914b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:10.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.186+0000 7f081914b640 10 monclient(hunting): picked mon.noname-a con 0x7f081406ee40 addr v1:172.21.15.18:6789/0 2024-03-29T11:19:10.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.186+0000 7f081914b640 10 monclient(hunting): picked mon.noname-b con 0x7f08140aeda0 addr v1:172.21.15.18:6790/0 2024-03-29T11:19:10.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.186+0000 7f081914b640 10 monclient(hunting): picked mon.noname-c con 0x7f081413c470 addr v1:172.21.15.70:6789/0 2024-03-29T11:19:10.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.186+0000 7f081914b640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f081406fdb0 con 0x7f081406ee40 2024-03-29T11:19:10.169 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.186+0000 7f081914b640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0814071090 con 0x7f081413c470 2024-03-29T11:19:10.170 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.189+0000 7f0812d9d640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f081406ee40 0x7f081413bd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:56182/0 (socket says 172.21.15.18:56182) 2024-03-29T11:19:10.170 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.189+0000 7f0812d9d640 1 -- 172.21.15.18:0/1703884428 learned_addr learned my addr 172.21.15.18:0/1703884428 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:19:10.170 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.189+0000 7f081914b640 1 -- 172.21.15.18:0/1703884428 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0814071370 con 0x7f08140aeda0 2024-03-29T11:19:10.170 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.189+0000 7f081914b640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:10.170 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.189+0000 7f081914b640 10 monclient(hunting): authenticate will time out at 1376.461093s 2024-03-29T11:19:10.170 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.190+0000 7f0811d9b640 1 -- 172.21.15.18:0/1703884428 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1379521778 0 0) 0x7f081406fdb0 con 0x7f081406ee40 2024-03-29T11:19:10.170 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.190+0000 7f0811d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:19:10.170 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.190+0000 7f0811d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:10.170 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.190+0000 7f0811d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:10.170 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.190+0000 7f0811d9b640 10 monclient(hunting): my global_id is 4560 2024-03-29T11:19:10.170 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.190+0000 7f0811d9b640 1 -- 172.21.15.18:0/1703884428 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07fc003520 con 0x7f081406ee40 2024-03-29T11:19:10.170 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.190+0000 7f0811d9b640 1 -- 172.21.15.18:0/1703884428 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1889047728 0 0) 0x7f0814071090 con 0x7f081413c470 2024-03-29T11:19:10.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.190+0000 7f0811d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:19:10.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.190+0000 7f0811d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:10.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.190+0000 7f0811d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:10.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.190+0000 7f0811d9b640 10 monclient(hunting): my global_id is 4519 2024-03-29T11:19:10.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.190+0000 7f0811d9b640 1 -- 172.21.15.18:0/1703884428 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f081406fdb0 con 0x7f081413c470 2024-03-29T11:19:10.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.190+0000 7f0811d9b640 1 -- 172.21.15.18:0/1703884428 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3895446641 0 0) 0x7f07fc003520 con 0x7f081406ee40 2024-03-29T11:19:10.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.190+0000 7f0811d9b640 1 -- 172.21.15.18:0/1703884428 >> v1:172.21.15.18:6790/0 conn(0x7f08140aeda0 legacy=0x7f0814071790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:10.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.191+0000 7f0811d9b640 1 -- 172.21.15.18:0/1703884428 >> v1:172.21.15.70:6789/0 conn(0x7f081413c470 legacy=0x7f081414a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:10.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.191+0000 7f0811d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:19:10.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.191+0000 7f0811d9b640 1 monclient: found mon.noname-a 2024-03-29T11:19:10.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.191+0000 7f0811d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:10.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.191+0000 7f0811d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-29T11:19:10.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.191+0000 7f0811d9b640 1 -- 172.21.15.18:0/1703884428 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0814070220 con 0x7f081406ee40 2024-03-29T11:19:10.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.191+0000 7f0811d9b640 10 monclient: _finish_auth 0 2024-03-29T11:19:10.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.191+0000 7f0811d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:19:10.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.191+0000 7f0811d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:10.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.191+0000 7f081914b640 5 monclient: authenticate success, global_id 4560 2024-03-29T11:19:10.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.191+0000 7f081914b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:10.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.191+0000 7f0811d9b640 1 -- 172.21.15.18:0/1703884428 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f0804001690 con 0x7f081406ee40 2024-03-29T11:19:10.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.191+0000 7f0811d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:10.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.191+0000 7f0811d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:10.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.191+0000 7f0811d9b640 10 monclient: dump: 2024-03-29T11:19:10.172 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:19:10.172 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:10.172 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:10.172 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:10.172 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:10.172 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:10.172 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:19:10.173 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:19:10.173 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:19:10.173 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:10.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.191+0000 7f081914b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:10.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.191+0000 7f0811d9b640 1 -- 172.21.15.18:0/1703884428 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08040018b0 con 0x7f081406ee40 2024-03-29T11:19:10.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.191+0000 7f0811d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:10.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.191+0000 7f081914b640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:10.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.191+0000 7f081914b640 4 set_mon_vals no callback set 2024-03-29T11:19:10.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.191+0000 7f0811d9b640 1 -- 172.21.15.18:0/1703884428 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f0804001c40 con 0x7f081406ee40 2024-03-29T11:19:10.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.191+0000 7f0811d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:10.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.191+0000 7f0811d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:10.173 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.191+0000 7f0811d9b640 10 monclient: dump: 2024-03-29T11:19:10.173 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:19:10.173 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:10.173 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:10.174 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:10.174 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:10.174 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:10.174 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:19:10.174 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:19:10.174 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:19:10.174 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:10.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.192+0000 7f081914b640 10 monclient: shutdown 2024-03-29T11:19:10.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.192+0000 7f081914b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:10.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.192+0000 7f081914b640 1 -- 172.21.15.18:0/1703884428 >> v1:172.21.15.18:6789/0 conn(0x7f081406ee40 legacy=0x7f081413bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:10.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.192+0000 7f081914b640 1 -- 172.21.15.18:0/1703884428 shutdown_connections 2024-03-29T11:19:10.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.192+0000 7f081914b640 1 -- 172.21.15.18:0/1703884428 >> 172.21.15.18:0/1703884428 conn(0x7f0814067470 msgr2=0x7f08140b48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:10.174 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.192+0000 7f081914b640 1 -- 172.21.15.18:0/1703884428 shutdown_connections 2024-03-29T11:19:10.175 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.192+0000 7f081914b640 1 -- 172.21.15.18:0/1703884428 wait complete. 2024-03-29T11:19:10.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.193+0000 7f081914b640 10 monclient: build_initial_monmap 2024-03-29T11:19:10.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.193+0000 7f081914b640 10 monclient: monmap: 2024-03-29T11:19:10.178 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:19:10.178 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:10.178 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:19:10.178 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:19:10.178 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:10.178 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:10.178 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:19:10.178 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:19:10.179 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:19:10.179 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:10.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.193+0000 7f081914b640 1 librados: starting msgr at 2024-03-29T11:19:10.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.193+0000 7f081914b640 1 librados: starting objecter 2024-03-29T11:19:10.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.194+0000 7f081914b640 1 Processor -- start 2024-03-29T11:19:10.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.194+0000 7f081914b640 1 -- start start 2024-03-29T11:19:10.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.194+0000 7f081914b640 1 librados: setting wanted keys 2024-03-29T11:19:10.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.194+0000 7f081914b640 1 librados: calling monclient init 2024-03-29T11:19:10.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.194+0000 7f081914b640 10 monclient: init 2024-03-29T11:19:10.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.195+0000 7f081914b640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:10.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.195+0000 7f081914b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:10.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.195+0000 7f081914b640 10 monclient(hunting): picked mon.noname-b con 0x7f081406ee40 addr v1:172.21.15.70:6789/0 2024-03-29T11:19:10.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.195+0000 7f081914b640 10 monclient(hunting): picked mon.noname-a con 0x7f08140aeda0 addr v1:172.21.15.18:6789/0 2024-03-29T11:19:10.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.196+0000 7f081914b640 10 monclient(hunting): picked mon.noname-c con 0x7f081413c470 addr v1:172.21.15.18:6790/0 2024-03-29T11:19:10.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.196+0000 7f081259c640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f08140aeda0 0x7f0814131b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:60322/0 (socket says 172.21.15.18:60322) 2024-03-29T11:19:10.180 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.196+0000 7f081259c640 1 -- 172.21.15.18:0/1863930041 learned_addr learned my addr 172.21.15.18:0/1863930041 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:19:10.180 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.196+0000 7f081914b640 1 -- 172.21.15.18:0/1863930041 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f081413dc70 con 0x7f08140aeda0 2024-03-29T11:19:10.180 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.196+0000 7f081914b640 1 -- 172.21.15.18:0/1863930041 --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08141354e0 con 0x7f081406ee40 2024-03-29T11:19:10.180 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.196+0000 7f081914b640 1 -- 172.21.15.18:0/1863930041 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0814133610 con 0x7f081413c470 2024-03-29T11:19:10.180 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.196+0000 7f081914b640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:10.180 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.196+0000 7f081914b640 10 monclient(hunting): authenticate will time out at 1376.468681s 2024-03-29T11:19:10.180 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.196+0000 7f07fb7fe640 10 client.?.objecter ms_handle_connect 0x7f08140aeda0 2024-03-29T11:19:10.180 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.196+0000 7f07fb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:10.180 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.197+0000 7f07fb7fe640 1 -- 172.21.15.18:0/1863930041 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 886520484 0 0) 0x7f081413dc70 con 0x7f08140aeda0 2024-03-29T11:19:10.180 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.197+0000 7f07fb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:10.180 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.197+0000 7f07fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:10.180 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.197+0000 7f07fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:10.180 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.197+0000 7f07fb7fe640 10 monclient(hunting): my global_id is 4563 2024-03-29T11:19:10.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.197+0000 7f07fb7fe640 1 -- 172.21.15.18:0/1863930041 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0800004eb0 con 0x7f08140aeda0 2024-03-29T11:19:10.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.197+0000 7f07fb7fe640 1 -- 172.21.15.18:0/1863930041 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2735412905 0 0) 0x7f0800004eb0 con 0x7f08140aeda0 2024-03-29T11:19:10.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.197+0000 7f07fb7fe640 1 -- 172.21.15.18:0/1863930041 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f081413dc70 con 0x7f08140aeda0 2024-03-29T11:19:10.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.197+0000 7f07fb7fe640 1 -- 172.21.15.18:0/1863930041 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f0808001990 con 0x7f08140aeda0 2024-03-29T11:19:10.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.197+0000 7f07fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:10.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.197+0000 7f07fb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:10.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.197+0000 7f07fb7fe640 10 monclient(hunting): dump: 2024-03-29T11:19:10.181 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:19:10.181 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:10.181 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:10.181 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:10.181 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:10.181 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:10.181 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:19:10.182 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:19:10.182 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:19:10.182 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:10.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.197+0000 7f07fb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:10.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.198+0000 7f07fb7fe640 1 -- 172.21.15.18:0/1863930041 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3863867979 0 0) 0x7f081413dc70 con 0x7f08140aeda0 2024-03-29T11:19:10.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.198+0000 7f07fb7fe640 1 -- 172.21.15.18:0/1863930041 >> v1:172.21.15.18:6790/0 conn(0x7f081413c470 legacy=0x7f08141de540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:19:10.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.198+0000 7f07fb7fe640 1 -- 172.21.15.18:0/1863930041 >> v1:172.21.15.70:6789/0 conn(0x7f081406ee40 legacy=0x7f0814133b30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:19:10.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.199+0000 7f07fb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:10.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.199+0000 7f07fb7fe640 1 monclient: found mon.a 2024-03-29T11:19:10.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.199+0000 7f07fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:10.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.199+0000 7f07fb7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:19:10.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.199+0000 7f07fb7fe640 1 -- 172.21.15.18:0/1863930041 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08141d6e30 con 0x7f08140aeda0 2024-03-29T11:19:10.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.199+0000 7f07fb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:10.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.199+0000 7f07fb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:10.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.199+0000 7f07fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:10.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.199+0000 7f081914b640 5 monclient: authenticate success, global_id 4563 2024-03-29T11:19:10.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.199+0000 7f081914b640 10 monclient: _renew_subs 2024-03-29T11:19:10.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.199+0000 7f081914b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:19:10.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.199+0000 7f081914b640 1 -- 172.21.15.18:0/1863930041 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08141d7080 con 0x7f08140aeda0 2024-03-29T11:19:10.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.200+0000 7f07fb7fe640 1 -- 172.21.15.18:0/1863930041 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0808001c90 con 0x7f08140aeda0 2024-03-29T11:19:10.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.200+0000 7f07fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:10.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.200+0000 7f07fb7fe640 1 -- 172.21.15.18:0/1863930041 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f0808003a60 con 0x7f08140aeda0 2024-03-29T11:19:10.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.200+0000 7f07fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:10.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.200+0000 7f07fb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:10.184 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.200+0000 7f07fb7fe640 10 monclient: dump: 2024-03-29T11:19:10.184 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:19:10.184 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:10.185 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:10.185 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:10.185 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:10.185 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:10.185 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:19:10.185 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:19:10.185 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:19:10.185 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:10.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.200+0000 7f081914b640 10 client.4563.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:10.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.200+0000 7f081914b640 10 monclient: _renew_subs 2024-03-29T11:19:10.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.200+0000 7f081914b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:19:10.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.200+0000 7f081914b640 1 -- 172.21.15.18:0/1863930041 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f08141332d0 con 0x7f08140aeda0 2024-03-29T11:19:10.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.200+0000 7f081914b640 1 librados: init done 2024-03-29T11:19:10.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.200+0000 7f081159a640 4 set_mon_vals no callback set 2024-03-29T11:19:10.185 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.202+0000 7f07f97fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:10.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.202+0000 7f07f97fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:10.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.203+0000 7f07f97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:19:10.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.203+0000 7f07fb7fe640 1 -- 172.21.15.18:0/1863930041 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f0808003bc0 con 0x7f08140aeda0 2024-03-29T11:19:10.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.203+0000 7f07f97fa640 1 -- 172.21.15.18:0/1863930041 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0814075f10 con 0x7f08140aeda0 2024-03-29T11:19:10.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.204+0000 7f07fb7fe640 1 -- 172.21.15.18:0/1863930041 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3599602952 0 0) 0x7f0808054880 con 0x7f08140aeda0 2024-03-29T11:19:10.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.204+0000 7f07fb7fe640 10 client.4563.objecter ms_dispatch 0x7f0814000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:19:10.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.204+0000 7f07fb7fe640 3 client.4563.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:19:10.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.204+0000 7f07fb7fe640 3 client.4563.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:19:10.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.204+0000 7f07fb7fe640 20 client.4563.objecter dump_active .. 0 homeless 2024-03-29T11:19:10.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.205+0000 7f0812d9d640 10 monclient: get_auth_request con 0x7f080006dce0 auth_method 0 2024-03-29T11:19:10.186 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.207+0000 7f07fb7fe640 10 client.4563.objecter ms_handle_connect 0x7f080006dce0 2024-03-29T11:19:10.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.211+0000 7f07fb7fe640 1 -- 172.21.15.18:0/1863930041 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f08080501c0 con 0x7f08140aeda0 2024-03-29T11:19:10.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.212+0000 7f07fb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:10.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.212+0000 7f07fb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:10.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.539+0000 7f07f97fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:10.517 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.539+0000 7f07f97fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:10.517 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.539+0000 7f07f97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:19:10.517 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.539+0000 7f07f97fa640 1 -- 172.21.15.18:0/1863930041 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08140008d0 con 0x7f08140aeda0 2024-03-29T11:19:10.517 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.540+0000 7f07fb7fe640 1 -- 172.21.15.18:0/1863930041 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2084344819) 0x7f08080522c0 con 0x7f08140aeda0 2024-03-29T11:19:10.517 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.540+0000 7f07fb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:10.517 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.540+0000 7f07fb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:10.518 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:19:10.518 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":11,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:19:08.996105+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:08.996105+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:10.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.544+0000 7f07f97fa640 10 librados: watch_flush enter 2024-03-29T11:19:10.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.544+0000 7f07f97fa640 10 librados: watch_flush exit 2024-03-29T11:19:10.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.544+0000 7f07f97fa640 20 client.4563.objecter shutdown clearing up homeless session... 2024-03-29T11:19:10.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.544+0000 7f07f97fa640 10 client.4563.objecter successfully canceled tick 2024-03-29T11:19:10.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.544+0000 7f07f97fa640 1 -- 172.21.15.18:0/1863930041 >> v1:172.21.15.70:6816/3101871562 conn(0x7f080006dce0 legacy=0x7f08000701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:10.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.544+0000 7f07f97fa640 10 monclient: shutdown 2024-03-29T11:19:10.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.544+0000 7f07f97fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:10.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.544+0000 7f07f97fa640 1 -- 172.21.15.18:0/1863930041 >> v1:172.21.15.18:6789/0 conn(0x7f08140aeda0 legacy=0x7f0814131b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:10.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.544+0000 7f07f97fa640 1 -- 172.21.15.18:0/1863930041 shutdown_connections 2024-03-29T11:19:10.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.544+0000 7f07f97fa640 1 -- 172.21.15.18:0/1863930041 >> 172.21.15.18:0/1863930041 conn(0x7f0814067470 msgr2=0x7f081413bb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:10.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.547+0000 7f07f97fa640 1 -- 172.21.15.18:0/1863930041 shutdown_connections 2024-03-29T11:19:10.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.547+0000 7f07f97fa640 1 -- 172.21.15.18:0/1863930041 wait complete. 2024-03-29T11:19:10.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:10.547+0000 7f07f97fa640 1 librados: shutdown 2024-03-29T11:19:10.541 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:17.347 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:17.500 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.521+0000 7f028ffff640 10 monclient: get_monmap_and_config 2024-03-29T11:19:17.500 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.521+0000 7f028ffff640 10 monclient: build_initial_monmap 2024-03-29T11:19:17.500 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.521+0000 7f028ffff640 10 monclient: monmap: 2024-03-29T11:19:17.500 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:19:17.500 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:17.500 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:19:17.522625+0000 2024-03-29T11:19:17.500 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:19:17.522625+0000 2024-03-29T11:19:17.500 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:17.500 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:17.500 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:19:17.500 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:19:17.501 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:19:17.501 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:17.501 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.522+0000 7f028ffff640 1 Processor -- start 2024-03-29T11:19:17.501 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.523+0000 7f028ffff640 1 -- start start 2024-03-29T11:19:17.501 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.523+0000 7f028ffff640 10 monclient: init 2024-03-29T11:19:17.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.524+0000 7f028ffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:17.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.524+0000 7f028ffff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:17.502 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.524+0000 7f028ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f0288034850 addr v1:172.21.15.18:6789/0 2024-03-29T11:19:17.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.524+0000 7f028ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f02880c51c0 addr v1:172.21.15.70:6789/0 2024-03-29T11:19:17.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.524+0000 7f028ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f02880c7cd0 addr v1:172.21.15.18:6790/0 2024-03-29T11:19:17.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.524+0000 7f028ffff640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0288029190 con 0x7f0288034850 2024-03-29T11:19:17.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.524+0000 7f028ffff640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02880293e0 con 0x7f02880c51c0 2024-03-29T11:19:17.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.524+0000 7f028ffff640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02880c8340 con 0x7f02880c7cd0 2024-03-29T11:19:17.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.524+0000 7f028ffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:17.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.524+0000 7f028ffff640 10 monclient(hunting): authenticate will time out at 1383.796194s 2024-03-29T11:19:17.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.524+0000 7f028f7fe640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f02880c7cd0 0x7f0288029f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:55022/0 (socket says 172.21.15.18:55022) 2024-03-29T11:19:17.503 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.524+0000 7f028f7fe640 1 -- 172.21.15.18:0/1698566175 learned_addr learned my addr 172.21.15.18:0/1698566175 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:19:17.504 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.526+0000 7f028dffb640 1 -- 172.21.15.18:0/1698566175 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2326726474 0 0) 0x7f02880c8340 con 0x7f02880c7cd0 2024-03-29T11:19:17.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.526+0000 7f028dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:17.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.526+0000 7f028dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:17.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.526+0000 7f028dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:17.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.526+0000 7f028dffb640 10 monclient(hunting): my global_id is 4658 2024-03-29T11:19:17.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.526+0000 7f028dffb640 1 -- 172.21.15.18:0/1698566175 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0278003520 con 0x7f02880c7cd0 2024-03-29T11:19:17.507 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.527+0000 7f028dffb640 1 -- 172.21.15.18:0/1698566175 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3220795381 0 0) 0x7f0288029190 con 0x7f0288034850 2024-03-29T11:19:17.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.527+0000 7f028dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:17.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.527+0000 7f028dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:17.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.527+0000 7f028dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:17.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.527+0000 7f028dffb640 10 monclient(hunting): my global_id is 4566 2024-03-29T11:19:17.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.527+0000 7f028dffb640 1 -- 172.21.15.18:0/1698566175 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02880c8340 con 0x7f0288034850 2024-03-29T11:19:17.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.527+0000 7f028dffb640 1 -- 172.21.15.18:0/1698566175 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1703254449 0 0) 0x7f02880c8340 con 0x7f0288034850 2024-03-29T11:19:17.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.527+0000 7f028dffb640 1 -- 172.21.15.18:0/1698566175 >> v1:172.21.15.18:6790/0 conn(0x7f02880c7cd0 legacy=0x7f0288029f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:17.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.527+0000 7f028dffb640 1 -- 172.21.15.18:0/1698566175 >> v1:172.21.15.70:6789/0 conn(0x7f02880c51c0 legacy=0x7f0288029800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:19:17.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.528+0000 7f028dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:17.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.528+0000 7f028dffb640 1 monclient: found mon.noname-a 2024-03-29T11:19:17.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.528+0000 7f028dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:17.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.528+0000 7f028dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-29T11:19:17.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.528+0000 7f028dffb640 1 -- 172.21.15.18:0/1698566175 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0288032980 con 0x7f0288034850 2024-03-29T11:19:17.508 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.528+0000 7f028dffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:17.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.528+0000 7f028dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:17.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.528+0000 7f028dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:17.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.528+0000 7f028dffb640 1 -- 172.21.15.18:0/1698566175 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f029004f730 con 0x7f0288034850 2024-03-29T11:19:17.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.528+0000 7f028dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:17.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.528+0000 7f028dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:17.509 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.528+0000 7f028dffb640 10 monclient: dump: 2024-03-29T11:19:17.509 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:19:17.509 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:17.509 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:17.510 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:17.510 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:17.510 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:17.510 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:19:17.510 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:19:17.510 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:19:17.510 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:17.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.528+0000 7f028dffb640 1 -- 172.21.15.18:0/1698566175 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f02900008d0 con 0x7f0288034850 2024-03-29T11:19:17.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.528+0000 7f028dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:17.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.528+0000 7f028dffb640 1 -- 172.21.15.18:0/1698566175 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f02900758b0 con 0x7f0288034850 2024-03-29T11:19:17.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.528+0000 7f028dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:17.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.528+0000 7f028dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:17.510 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.528+0000 7f028dffb640 10 monclient: dump: 2024-03-29T11:19:17.510 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:19:17.511 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:17.511 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:17.511 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:17.511 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:17.511 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:17.511 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:19:17.511 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:19:17.511 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:19:17.511 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:17.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.528+0000 7f028ffff640 5 monclient: authenticate success, global_id 4566 2024-03-29T11:19:17.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.528+0000 7f028ffff640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:17.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.528+0000 7f028ffff640 4 set_mon_vals no callback set 2024-03-29T11:19:17.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.529+0000 7f028ffff640 10 monclient: shutdown 2024-03-29T11:19:17.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.529+0000 7f028ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:17.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.529+0000 7f028ffff640 1 -- 172.21.15.18:0/1698566175 >> v1:172.21.15.18:6789/0 conn(0x7f0288034850 legacy=0x7f02880c4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:17.511 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.529+0000 7f028ffff640 1 -- 172.21.15.18:0/1698566175 shutdown_connections 2024-03-29T11:19:17.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.529+0000 7f028ffff640 1 -- 172.21.15.18:0/1698566175 >> 172.21.15.18:0/1698566175 conn(0x7f02880058f0 msgr2=0x7f0288005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:17.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.529+0000 7f028ffff640 1 -- 172.21.15.18:0/1698566175 shutdown_connections 2024-03-29T11:19:17.512 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.529+0000 7f028ffff640 1 -- 172.21.15.18:0/1698566175 wait complete. 2024-03-29T11:19:17.513 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.530+0000 7f028ffff640 10 monclient: build_initial_monmap 2024-03-29T11:19:17.513 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.530+0000 7f028ffff640 10 monclient: monmap: 2024-03-29T11:19:17.513 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:19:17.513 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:17.513 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:19:17.514 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:19:17.514 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:17.514 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:17.514 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:19:17.514 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:19:17.514 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:19:17.514 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:17.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.531+0000 7f028ffff640 1 librados: starting msgr at 2024-03-29T11:19:17.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.531+0000 7f028ffff640 1 librados: starting objecter 2024-03-29T11:19:17.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.531+0000 7f028ffff640 1 Processor -- start 2024-03-29T11:19:17.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.531+0000 7f028ffff640 1 -- start start 2024-03-29T11:19:17.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.531+0000 7f028ffff640 1 librados: setting wanted keys 2024-03-29T11:19:17.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.531+0000 7f028ffff640 1 librados: calling monclient init 2024-03-29T11:19:17.514 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.531+0000 7f028ffff640 10 monclient: init 2024-03-29T11:19:17.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.532+0000 7f028ffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:17.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.532+0000 7f028ffff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:19:17.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.532+0000 7f028ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f0288034850 addr v1:172.21.15.18:6790/0 2024-03-29T11:19:17.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.532+0000 7f028ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f02880c51c0 addr v1:172.21.15.18:6789/0 2024-03-29T11:19:17.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.533+0000 7f028effd640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f0288034850 0x7f02880ca1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:58200/0 (socket says 172.21.15.18:58200) 2024-03-29T11:19:17.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.534+0000 7f028ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f02880c7cd0 addr v1:172.21.15.70:6789/0 2024-03-29T11:19:17.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.534+0000 7f028ffff640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02880b4170 con 0x7f02880c51c0 2024-03-29T11:19:17.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.534+0000 7f028ffff640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02880b43a0 con 0x7f02880c7cd0 2024-03-29T11:19:17.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.534+0000 7f028ffff640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02880c9020 con 0x7f0288034850 2024-03-29T11:19:17.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.534+0000 7f028ffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:17.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.534+0000 7f028ffff640 10 monclient(hunting): authenticate will time out at 1383.806562s 2024-03-29T11:19:17.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.535+0000 7f028e7fc640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f02880c51c0 0x7f02880c9540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:58003/0 (socket says 172.21.15.18:58003) 2024-03-29T11:19:17.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.535+0000 7f028e7fc640 1 -- 172.21.15.18:0/901732145 learned_addr learned my addr 172.21.15.18:0/901732145 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:19:17.515 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.535+0000 7f02777fe640 10 client.?.objecter ms_handle_connect 0x7f02880c51c0 2024-03-29T11:19:17.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.535+0000 7f02777fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:17.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.536+0000 7f02777fe640 1 -- 172.21.15.18:0/901732145 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3676687127 0 0) 0x7f02880b4170 con 0x7f02880c51c0 2024-03-29T11:19:17.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.536+0000 7f02777fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:17.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.536+0000 7f02777fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:17.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.536+0000 7f02777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:17.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.536+0000 7f02777fe640 10 monclient(hunting): my global_id is 4569 2024-03-29T11:19:17.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.536+0000 7f02777fe640 1 -- 172.21.15.18:0/901732145 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f027c004eb0 con 0x7f02880c51c0 2024-03-29T11:19:17.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.536+0000 7f02777fe640 10 client.?.objecter ms_handle_connect 0x7f0288034850 2024-03-29T11:19:17.516 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.536+0000 7f02777fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:17.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.537+0000 7f02777fe640 1 -- 172.21.15.18:0/901732145 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3038983920 0 0) 0x7f02880c9020 con 0x7f0288034850 2024-03-29T11:19:17.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.537+0000 7f02777fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:17.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.537+0000 7f02777fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:17.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.537+0000 7f02777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:17.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.537+0000 7f02777fe640 10 monclient(hunting): my global_id is 4661 2024-03-29T11:19:17.519 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.537+0000 7f02777fe640 1 -- 172.21.15.18:0/901732145 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f027c0051d0 con 0x7f0288034850 2024-03-29T11:19:17.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.537+0000 7f02777fe640 1 -- 172.21.15.18:0/901732145 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 743252795 0 0) 0x7f027c0051d0 con 0x7f0288034850 2024-03-29T11:19:17.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.537+0000 7f02777fe640 1 -- 172.21.15.18:0/901732145 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f02880c9020 con 0x7f0288034850 2024-03-29T11:19:17.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.537+0000 7f02777fe640 1 -- 172.21.15.18:0/901732145 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f029004f400 con 0x7f0288034850 2024-03-29T11:19:17.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.537+0000 7f02777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:17.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.537+0000 7f02777fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:17.520 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.537+0000 7f02777fe640 10 monclient(hunting): dump: 2024-03-29T11:19:17.520 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:19:17.520 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:17.520 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:17.520 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:17.520 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:17.521 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:17.521 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:19:17.521 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:19:17.521 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:19:17.521 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:17.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.537+0000 7f02777fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:17.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.538+0000 7f02777fe640 1 -- 172.21.15.18:0/901732145 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1085084720 0 0) 0x7f02880c9020 con 0x7f0288034850 2024-03-29T11:19:17.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.538+0000 7f02777fe640 1 -- 172.21.15.18:0/901732145 >> v1:172.21.15.70:6789/0 conn(0x7f02880c7cd0 legacy=0x7f02880b2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:17.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.538+0000 7f02777fe640 1 -- 172.21.15.18:0/901732145 >> v1:172.21.15.18:6789/0 conn(0x7f02880c51c0 legacy=0x7f02880c9540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:17.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.538+0000 7f02777fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:17.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.538+0000 7f02777fe640 1 monclient: found mon.c 2024-03-29T11:19:17.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.538+0000 7f02777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:17.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.538+0000 7f02777fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:19:17.521 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.538+0000 7f02777fe640 1 -- 172.21.15.18:0/901732145 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02880b8050 con 0x7f0288034850 2024-03-29T11:19:17.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.538+0000 7f02777fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:17.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.538+0000 7f02777fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:17.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.538+0000 7f02777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:17.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.538+0000 7f028ffff640 5 monclient: authenticate success, global_id 4661 2024-03-29T11:19:17.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.538+0000 7f028ffff640 10 monclient: _renew_subs 2024-03-29T11:19:17.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.538+0000 7f028ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:19:17.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.538+0000 7f028ffff640 1 -- 172.21.15.18:0/901732145 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02880c8b90 con 0x7f0288034850 2024-03-29T11:19:17.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.538+0000 7f028ffff640 10 client.4661.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:17.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.538+0000 7f028ffff640 10 monclient: _renew_subs 2024-03-29T11:19:17.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.538+0000 7f028ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:19:17.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.538+0000 7f028ffff640 1 -- 172.21.15.18:0/901732145 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f02880b3760 con 0x7f0288034850 2024-03-29T11:19:17.522 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.538+0000 7f028ffff640 1 librados: init done 2024-03-29T11:19:17.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.539+0000 7f02757fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:17.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.539+0000 7f02757fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:17.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.539+0000 7f02757fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:19:17.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.539+0000 7f02757fa640 1 -- 172.21.15.18:0/901732145 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0270005080 con 0x7f0288034850 2024-03-29T11:19:17.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.540+0000 7f02777fe640 1 -- 172.21.15.18:0/901732145 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f029004f730 con 0x7f0288034850 2024-03-29T11:19:17.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.540+0000 7f02777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:17.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.540+0000 7f02777fe640 1 -- 172.21.15.18:0/901732145 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f0290067d90 con 0x7f0288034850 2024-03-29T11:19:17.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.540+0000 7f02777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:17.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.540+0000 7f02777fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:17.523 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.540+0000 7f02777fe640 10 monclient: dump: 2024-03-29T11:19:17.524 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:19:17.524 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:17.524 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:17.524 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:17.524 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:17.524 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:17.524 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:19:17.524 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:19:17.524 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:19:17.524 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:17.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.540+0000 7f02777fe640 1 -- 172.21.15.18:0/901732145 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f0290067fb0 con 0x7f0288034850 2024-03-29T11:19:17.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.541+0000 7f028d7fa640 4 set_mon_vals no callback set 2024-03-29T11:19:17.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.541+0000 7f02777fe640 1 -- 172.21.15.18:0/901732145 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3599602952 0 0) 0x7f02900ff4c0 con 0x7f0288034850 2024-03-29T11:19:17.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.541+0000 7f02777fe640 10 client.4661.objecter ms_dispatch 0x7f0290000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:19:17.524 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.541+0000 7f02777fe640 3 client.4661.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:19:17.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.541+0000 7f02777fe640 3 client.4661.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:19:17.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.541+0000 7f02777fe640 20 client.4661.objecter dump_active .. 0 homeless 2024-03-29T11:19:17.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.541+0000 7f028e7fc640 10 monclient: get_auth_request con 0x7f027c06de60 auth_method 0 2024-03-29T11:19:17.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.542+0000 7f02777fe640 10 client.4661.objecter ms_handle_connect 0x7f027c06de60 2024-03-29T11:19:17.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.548+0000 7f02777fe640 1 -- 172.21.15.18:0/901732145 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f02900caf10 con 0x7f0288034850 2024-03-29T11:19:17.525 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.549+0000 7f02777fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:17.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.549+0000 7f02777fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:17.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.886+0000 7f02757fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:17.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.886+0000 7f02757fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:17.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.886+0000 7f02757fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:19:17.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.886+0000 7f02757fa640 1 -- 172.21.15.18:0/901732145 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0270005330 con 0x7f0288034850 2024-03-29T11:19:17.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.887+0000 7f02777fe640 1 -- 172.21.15.18:0/901732145 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2084344819) 0x7f02900d7330 con 0x7f0288034850 2024-03-29T11:19:17.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.887+0000 7f02777fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:17.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.887+0000 7f02777fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:17.865 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:19:17.865 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":11,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:19:08.996105+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:08.996105+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:17.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.891+0000 7f02757fa640 10 librados: watch_flush enter 2024-03-29T11:19:17.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.891+0000 7f02757fa640 10 librados: watch_flush exit 2024-03-29T11:19:17.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.891+0000 7f02757fa640 20 client.4661.objecter shutdown clearing up homeless session... 2024-03-29T11:19:17.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.891+0000 7f02757fa640 10 client.4661.objecter successfully canceled tick 2024-03-29T11:19:17.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.891+0000 7f02757fa640 1 -- 172.21.15.18:0/901732145 >> v1:172.21.15.70:6816/3101871562 conn(0x7f027c06de60 legacy=0x7f027c070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:17.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.891+0000 7f02757fa640 10 monclient: shutdown 2024-03-29T11:19:17.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.891+0000 7f02757fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:17.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.891+0000 7f02757fa640 1 -- 172.21.15.18:0/901732145 >> v1:172.21.15.18:6790/0 conn(0x7f0288034850 legacy=0x7f02880ca1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:17.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.892+0000 7f02757fa640 1 -- 172.21.15.18:0/901732145 shutdown_connections 2024-03-29T11:19:17.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.892+0000 7f02757fa640 1 -- 172.21.15.18:0/901732145 >> 172.21.15.18:0/901732145 conn(0x7f02880058f0 msgr2=0x7f02880c1a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:17.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.892+0000 7f02757fa640 1 -- 172.21.15.18:0/901732145 shutdown_connections 2024-03-29T11:19:17.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.892+0000 7f02757fa640 1 -- 172.21.15.18:0/901732145 wait complete. 2024-03-29T11:19:17.871 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:17.893+0000 7f02757fa640 1 librados: shutdown 2024-03-29T11:19:17.888 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:24.694 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:24.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.864+0000 7f177a84a640 10 monclient: get_monmap_and_config 2024-03-29T11:19:24.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.864+0000 7f177a84a640 10 monclient: build_initial_monmap 2024-03-29T11:19:24.841 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.864+0000 7f177a84a640 10 monclient: monmap: 2024-03-29T11:19:24.841 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:19:24.841 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:24.841 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:19:24.865335+0000 2024-03-29T11:19:24.841 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:19:24.865335+0000 2024-03-29T11:19:24.842 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:24.842 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:24.842 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:19:24.842 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:19:24.842 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:19:24.842 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:24.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.865+0000 7f177a84a640 1 Processor -- start 2024-03-29T11:19:24.842 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.865+0000 7f177a84a640 1 -- start start 2024-03-29T11:19:24.843 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.865+0000 7f177a84a640 10 monclient: init 2024-03-29T11:19:24.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.866+0000 7f177a84a640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:24.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.866+0000 7f177a84a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:24.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.866+0000 7f177a84a640 10 monclient(hunting): picked mon.noname-c con 0x7f17741422b0 addr v1:172.21.15.70:6789/0 2024-03-29T11:19:24.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.866+0000 7f177a84a640 10 monclient(hunting): picked mon.noname-b con 0x7f1774142690 addr v1:172.21.15.18:6790/0 2024-03-29T11:19:24.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.866+0000 7f177a84a640 10 monclient(hunting): picked mon.noname-a con 0x7f17740b17d0 addr v1:172.21.15.18:6789/0 2024-03-29T11:19:24.846 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.866+0000 7f177a84a640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1774142c80 con 0x7f17740b17d0 2024-03-29T11:19:24.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.866+0000 7f177a84a640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17740aae40 con 0x7f17741422b0 2024-03-29T11:19:24.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.866+0000 7f177a84a640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f177413fbd0 con 0x7f1774142690 2024-03-29T11:19:24.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.866+0000 7f177a84a640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:24.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.866+0000 7f177a84a640 10 monclient(hunting): authenticate will time out at 1391.138635s 2024-03-29T11:19:24.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.866+0000 7f17737fe640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f1774142690 0x7f17740b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:50194/0 (socket says 172.21.15.18:50194) 2024-03-29T11:19:24.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.866+0000 7f17737fe640 1 -- 172.21.15.18:0/901157720 learned_addr learned my addr 172.21.15.18:0/901157720 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:19:24.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.867+0000 7f1772ffd640 1 -- 172.21.15.18:0/901157720 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3289314722 0 0) 0x7f1774142c80 con 0x7f17740b17d0 2024-03-29T11:19:24.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.867+0000 7f1772ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:19:24.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.867+0000 7f1772ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:24.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.867+0000 7f1772ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:24.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.867+0000 7f1772ffd640 10 monclient(hunting): my global_id is 4572 2024-03-29T11:19:24.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.867+0000 7f1772ffd640 1 -- 172.21.15.18:0/901157720 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f175c003520 con 0x7f17740b17d0 2024-03-29T11:19:24.847 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.868+0000 7f1772ffd640 1 -- 172.21.15.18:0/901157720 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1135645790 0 0) 0x7f175c003520 con 0x7f17740b17d0 2024-03-29T11:19:24.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.868+0000 7f1772ffd640 1 -- 172.21.15.18:0/901157720 >> v1:172.21.15.18:6790/0 conn(0x7f1774142690 legacy=0x7f17740b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:24.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.868+0000 7f1772ffd640 1 -- 172.21.15.18:0/901157720 >> v1:172.21.15.70:6789/0 conn(0x7f17741422b0 legacy=0x7f177413f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:19:24.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.868+0000 7f1772ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:19:24.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.868+0000 7f1772ffd640 1 monclient: found mon.noname-a 2024-03-29T11:19:24.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.868+0000 7f1772ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:24.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.868+0000 7f1772ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-29T11:19:24.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.868+0000 7f1772ffd640 1 -- 172.21.15.18:0/901157720 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1774151940 con 0x7f17740b17d0 2024-03-29T11:19:24.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.868+0000 7f1772ffd640 10 monclient: _finish_auth 0 2024-03-29T11:19:24.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.868+0000 7f1772ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:19:24.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.868+0000 7f1772ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:24.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.868+0000 7f1772ffd640 1 -- 172.21.15.18:0/901157720 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f176c0019f0 con 0x7f17740b17d0 2024-03-29T11:19:24.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.868+0000 7f1772ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:24.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.868+0000 7f1772ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:24.848 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.868+0000 7f1772ffd640 10 monclient: dump: 2024-03-29T11:19:24.849 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:19:24.849 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:24.849 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:24.849 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:24.849 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:24.849 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:24.849 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:19:24.849 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:19:24.849 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:19:24.849 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:24.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.868+0000 7f177a84a640 5 monclient: authenticate success, global_id 4572 2024-03-29T11:19:24.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.868+0000 7f177a84a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:24.849 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.868+0000 7f1772ffd640 1 -- 172.21.15.18:0/901157720 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f176c001c10 con 0x7f17740b17d0 2024-03-29T11:19:24.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.868+0000 7f1772ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:24.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.868+0000 7f177a84a640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:24.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.868+0000 7f177a84a640 4 set_mon_vals no callback set 2024-03-29T11:19:24.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.868+0000 7f1772ffd640 1 -- 172.21.15.18:0/901157720 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f176c001fe0 con 0x7f17740b17d0 2024-03-29T11:19:24.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.868+0000 7f1772ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:24.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.868+0000 7f1772ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:24.850 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.868+0000 7f1772ffd640 10 monclient: dump: 2024-03-29T11:19:24.850 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:19:24.850 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:24.850 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:24.850 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:24.850 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:24.850 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:24.851 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:19:24.851 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:19:24.851 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:19:24.851 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:24.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.869+0000 7f177a84a640 10 monclient: shutdown 2024-03-29T11:19:24.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.869+0000 7f177a84a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:24.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.869+0000 7f177a84a640 1 -- 172.21.15.18:0/901157720 >> v1:172.21.15.18:6789/0 conn(0x7f17740b17d0 legacy=0x7f17740b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:24.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.869+0000 7f177a84a640 1 -- 172.21.15.18:0/901157720 shutdown_connections 2024-03-29T11:19:24.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.869+0000 7f177a84a640 1 -- 172.21.15.18:0/901157720 >> 172.21.15.18:0/901157720 conn(0x7f1774066a00 msgr2=0x7f1774066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:24.851 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.869+0000 7f177a84a640 1 -- 172.21.15.18:0/901157720 shutdown_connections 2024-03-29T11:19:24.858 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.869+0000 7f177a84a640 1 -- 172.21.15.18:0/901157720 wait complete. 2024-03-29T11:19:24.858 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.870+0000 7f177a84a640 10 monclient: build_initial_monmap 2024-03-29T11:19:24.858 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.870+0000 7f177a84a640 10 monclient: monmap: 2024-03-29T11:19:24.858 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:19:24.858 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:24.858 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:19:24.858 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:19:24.858 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:24.858 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:24.858 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:19:24.858 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:19:24.859 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:19:24.859 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:24.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.871+0000 7f177a84a640 1 librados: starting msgr at 2024-03-29T11:19:24.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.871+0000 7f177a84a640 1 librados: starting objecter 2024-03-29T11:19:24.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.871+0000 7f177a84a640 1 Processor -- start 2024-03-29T11:19:24.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.871+0000 7f177a84a640 1 -- start start 2024-03-29T11:19:24.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.871+0000 7f177a84a640 1 librados: setting wanted keys 2024-03-29T11:19:24.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.871+0000 7f177a84a640 1 librados: calling monclient init 2024-03-29T11:19:24.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.871+0000 7f177a84a640 10 monclient: init 2024-03-29T11:19:24.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.872+0000 7f177a84a640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:24.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.872+0000 7f177a84a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:19:24.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.872+0000 7f177a84a640 10 monclient(hunting): picked mon.noname-b con 0x7f17740b17d0 addr v1:172.21.15.70:6789/0 2024-03-29T11:19:24.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.872+0000 7f177a84a640 10 monclient(hunting): picked mon.noname-a con 0x7f17741422b0 addr v1:172.21.15.18:6789/0 2024-03-29T11:19:24.859 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.872+0000 7f177a84a640 10 monclient(hunting): picked mon.noname-c con 0x7f1774142690 addr v1:172.21.15.18:6790/0 2024-03-29T11:19:24.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.872+0000 7f177a84a640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17740b1030 con 0x7f17741422b0 2024-03-29T11:19:24.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.872+0000 7f177a84a640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17740b12a0 con 0x7f17740b17d0 2024-03-29T11:19:24.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.872+0000 7f177a84a640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17740b04f0 con 0x7f1774142690 2024-03-29T11:19:24.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.872+0000 7f177a84a640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:24.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.872+0000 7f177a84a640 10 monclient(hunting): authenticate will time out at 1391.144560s 2024-03-29T11:19:24.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.872+0000 7f1773fff640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f17740b17d0 0x7f17740af190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.18:52994/0 (socket says 172.21.15.18:52994) 2024-03-29T11:19:24.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.872+0000 7f1773fff640 1 -- 172.21.15.18:0/1070107838 learned_addr learned my addr 172.21.15.18:0/1070107838 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:19:24.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.872+0000 7f1778de7640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f1774142690 0x7f17741e1eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:51506/0 (socket says 172.21.15.18:51506) 2024-03-29T11:19:24.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.873+0000 7f1770ff9640 10 client.?.objecter ms_handle_connect 0x7f17740b17d0 2024-03-29T11:19:24.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.873+0000 7f1770ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:24.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.873+0000 7f1770ff9640 10 client.?.objecter ms_handle_connect 0x7f1774142690 2024-03-29T11:19:24.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.873+0000 7f1770ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:24.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.873+0000 7f1770ff9640 10 client.?.objecter ms_handle_connect 0x7f17741422b0 2024-03-29T11:19:24.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.873+0000 7f1770ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:24.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.873+0000 7f1770ff9640 1 -- 172.21.15.18:0/1070107838 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2332610817 0 0) 0x7f17740b12a0 con 0x7f17740b17d0 2024-03-29T11:19:24.860 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.873+0000 7f1770ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:24.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.873+0000 7f1770ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:24.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.873+0000 7f1770ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:24.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.873+0000 7f1770ff9640 10 monclient(hunting): my global_id is 4522 2024-03-29T11:19:24.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.873+0000 7f1770ff9640 1 -- 172.21.15.18:0/1070107838 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1760004eb0 con 0x7f17740b17d0 2024-03-29T11:19:24.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.873+0000 7f1770ff9640 1 -- 172.21.15.18:0/1070107838 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1798128722 0 0) 0x7f17740b04f0 con 0x7f1774142690 2024-03-29T11:19:24.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.873+0000 7f1770ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:24.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.873+0000 7f1770ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:24.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.873+0000 7f1770ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:24.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.873+0000 7f1770ff9640 10 monclient(hunting): my global_id is 4667 2024-03-29T11:19:24.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.873+0000 7f1770ff9640 1 -- 172.21.15.18:0/1070107838 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17740b12a0 con 0x7f1774142690 2024-03-29T11:19:24.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.873+0000 7f1770ff9640 1 -- 172.21.15.18:0/1070107838 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 392751772 0 0) 0x7f17740b1030 con 0x7f17741422b0 2024-03-29T11:19:24.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.873+0000 7f1770ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:19:24.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.873+0000 7f1770ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:24.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.873+0000 7f1770ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:24.861 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.873+0000 7f1770ff9640 10 monclient(hunting): my global_id is 4575 2024-03-29T11:19:24.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.873+0000 7f1770ff9640 1 -- 172.21.15.18:0/1070107838 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17740b04f0 con 0x7f17741422b0 2024-03-29T11:19:24.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.874+0000 7f1770ff9640 1 -- 172.21.15.18:0/1070107838 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3984429954 0 0) 0x7f17740b12a0 con 0x7f1774142690 2024-03-29T11:19:24.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.874+0000 7f1770ff9640 1 -- 172.21.15.18:0/1070107838 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17740b1030 con 0x7f1774142690 2024-03-29T11:19:24.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.874+0000 7f1770ff9640 1 -- 172.21.15.18:0/1070107838 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f176c001f20 con 0x7f1774142690 2024-03-29T11:19:24.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.874+0000 7f1770ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:24.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.874+0000 7f1770ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:24.862 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.874+0000 7f1770ff9640 10 monclient(hunting): dump: 2024-03-29T11:19:24.862 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:19:24.862 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:24.862 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:24.862 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:24.862 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:24.862 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:24.862 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:19:24.862 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:19:24.863 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:19:24.863 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:24.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.874+0000 7f1770ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:24.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.874+0000 7f1770ff9640 1 -- 172.21.15.18:0/1070107838 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3507779240 0 0) 0x7f17740b04f0 con 0x7f17741422b0 2024-03-29T11:19:24.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.874+0000 7f1770ff9640 1 -- 172.21.15.18:0/1070107838 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f17740b12a0 con 0x7f17741422b0 2024-03-29T11:19:24.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.874+0000 7f1770ff9640 1 -- 172.21.15.18:0/1070107838 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f1764001660 con 0x7f17741422b0 2024-03-29T11:19:24.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.874+0000 7f1770ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:24.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.874+0000 7f1770ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:24.863 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.874+0000 7f1770ff9640 10 monclient(hunting): dump: 2024-03-29T11:19:24.863 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:19:24.863 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:24.863 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:24.863 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:24.863 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:24.863 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:24.863 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:19:24.863 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:19:24.864 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:19:24.864 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:24.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.875+0000 7f1770ff9640 1 -- 172.21.15.18:0/1070107838 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2888971043 0 0) 0x7f17740b12a0 con 0x7f17741422b0 2024-03-29T11:19:24.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.875+0000 7f1770ff9640 1 -- 172.21.15.18:0/1070107838 >> v1:172.21.15.18:6790/0 conn(0x7f1774142690 legacy=0x7f17741e1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:24.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.875+0000 7f1770ff9640 1 -- 172.21.15.18:0/1070107838 >> v1:172.21.15.70:6789/0 conn(0x7f17740b17d0 legacy=0x7f17740af190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:24.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.875+0000 7f1770ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:19:24.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.875+0000 7f1770ff9640 1 monclient: found mon.a 2024-03-29T11:19:24.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.875+0000 7f1770ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:24.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.875+0000 7f1770ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:19:24.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.875+0000 7f1770ff9640 1 -- 172.21.15.18:0/1070107838 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17740afea0 con 0x7f17741422b0 2024-03-29T11:19:24.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.875+0000 7f1770ff9640 10 monclient: _finish_auth 0 2024-03-29T11:19:24.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.875+0000 7f1770ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:19:24.864 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.875+0000 7f1770ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:24.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.875+0000 7f177a84a640 5 monclient: authenticate success, global_id 4575 2024-03-29T11:19:24.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.875+0000 7f177a84a640 10 monclient: _renew_subs 2024-03-29T11:19:24.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.875+0000 7f177a84a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:19:24.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.875+0000 7f177a84a640 1 -- 172.21.15.18:0/1070107838 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17740b0150 con 0x7f17741422b0 2024-03-29T11:19:24.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.875+0000 7f177a84a640 10 client.4575.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:24.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.875+0000 7f177a84a640 10 monclient: _renew_subs 2024-03-29T11:19:24.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.875+0000 7f177a84a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:19:24.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.875+0000 7f177a84a640 1 -- 172.21.15.18:0/1070107838 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f17741db4d0 con 0x7f17741422b0 2024-03-29T11:19:24.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.875+0000 7f177a84a640 1 librados: init done 2024-03-29T11:19:24.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.875+0000 7f1770ff9640 1 -- 172.21.15.18:0/1070107838 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f17640017c0 con 0x7f17741422b0 2024-03-29T11:19:24.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.875+0000 7f1770ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:24.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.875+0000 7f1770ff9640 1 -- 172.21.15.18:0/1070107838 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f17640047e0 con 0x7f17741422b0 2024-03-29T11:19:24.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.875+0000 7f1770ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:24.865 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.875+0000 7f1770ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:24.866 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.875+0000 7f1770ff9640 10 monclient: dump: 2024-03-29T11:19:24.866 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:19:24.866 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:24.866 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:24.866 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:24.866 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:24.866 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:24.866 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:19:24.866 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:19:24.866 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:19:24.866 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:24.866 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.875+0000 7f17727fc640 4 set_mon_vals no callback set 2024-03-29T11:19:24.866 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.876+0000 7f177a84a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:24.866 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.876+0000 7f177a84a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:24.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.876+0000 7f177a84a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:19:24.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.876+0000 7f177a84a640 1 -- 172.21.15.18:0/1070107838 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1754005080 con 0x7f17741422b0 2024-03-29T11:19:24.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.877+0000 7f1770ff9640 1 -- 172.21.15.18:0/1070107838 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f176401a7b0 con 0x7f17741422b0 2024-03-29T11:19:24.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.878+0000 7f1773fff640 10 monclient: get_auth_request con 0x7f176006df70 auth_method 0 2024-03-29T11:19:24.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.878+0000 7f1770ff9640 1 -- 172.21.15.18:0/1070107838 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3599602952 0 0) 0x7f1764084870 con 0x7f17741422b0 2024-03-29T11:19:24.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.878+0000 7f1770ff9640 10 client.4575.objecter ms_dispatch 0x7f1774000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:19:24.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.878+0000 7f1770ff9640 3 client.4575.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:19:24.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.878+0000 7f1770ff9640 3 client.4575.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:19:24.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.878+0000 7f1770ff9640 20 client.4575.objecter dump_active .. 0 homeless 2024-03-29T11:19:24.867 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.878+0000 7f1770ff9640 10 client.4575.objecter ms_handle_connect 0x7f176006df70 2024-03-29T11:19:24.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.885+0000 7f1770ff9640 1 -- 172.21.15.18:0/1070107838 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f176404ff80 con 0x7f17741422b0 2024-03-29T11:19:24.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.885+0000 7f1770ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:24.868 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:24.885+0000 7f1770ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:25.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:25.213+0000 7f177a84a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:25.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:25.213+0000 7f177a84a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:25.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:25.213+0000 7f177a84a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:19:25.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:25.213+0000 7f177a84a640 1 -- 172.21.15.18:0/1070107838 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1754005330 con 0x7f17741422b0 2024-03-29T11:19:25.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:25.213+0000 7f1770ff9640 1 -- 172.21.15.18:0/1070107838 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2084344819) 0x7f1764052080 con 0x7f17741422b0 2024-03-29T11:19:25.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:25.214+0000 7f1770ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:25.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:25.214+0000 7f1770ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:25.191 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:19:25.191 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":11,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:19:08.996105+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:08.996105+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:25.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:25.217+0000 7f177a84a640 10 librados: watch_flush enter 2024-03-29T11:19:25.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:25.217+0000 7f177a84a640 10 librados: watch_flush exit 2024-03-29T11:19:25.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:25.217+0000 7f177a84a640 20 client.4575.objecter shutdown clearing up homeless session... 2024-03-29T11:19:25.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:25.217+0000 7f177a84a640 10 client.4575.objecter successfully canceled tick 2024-03-29T11:19:25.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:25.217+0000 7f177a84a640 1 -- 172.21.15.18:0/1070107838 >> v1:172.21.15.70:6816/3101871562 conn(0x7f176006df70 legacy=0x7f1760070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:25.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:25.217+0000 7f177a84a640 10 monclient: shutdown 2024-03-29T11:19:25.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:25.217+0000 7f177a84a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:25.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:25.217+0000 7f177a84a640 1 -- 172.21.15.18:0/1070107838 >> v1:172.21.15.18:6789/0 conn(0x7f17741422b0 legacy=0x7f17741dabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:25.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:25.217+0000 7f177a84a640 1 -- 172.21.15.18:0/1070107838 shutdown_connections 2024-03-29T11:19:25.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:25.217+0000 7f177a84a640 1 -- 172.21.15.18:0/1070107838 >> 172.21.15.18:0/1070107838 conn(0x7f1774066a00 msgr2=0x7f17740b3ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:25.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:25.217+0000 7f177a84a640 1 -- 172.21.15.18:0/1070107838 shutdown_connections 2024-03-29T11:19:25.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:25.217+0000 7f177a84a640 1 -- 172.21.15.18:0/1070107838 wait complete. 2024-03-29T11:19:25.195 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:25.217+0000 7f177a84a640 1 librados: shutdown 2024-03-29T11:19:25.208 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:32.016 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:32.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.188+0000 7f72e1900640 10 monclient: get_monmap_and_config 2024-03-29T11:19:32.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.188+0000 7f72e1900640 10 monclient: build_initial_monmap 2024-03-29T11:19:32.165 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.188+0000 7f72e1900640 10 monclient: monmap: 2024-03-29T11:19:32.166 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:19:32.166 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:32.166 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:19:32.189014+0000 2024-03-29T11:19:32.166 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:19:32.189014+0000 2024-03-29T11:19:32.166 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:32.166 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:32.166 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:19:32.166 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:19:32.166 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:19:32.166 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:32.166 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.189+0000 7f72e1900640 1 Processor -- start 2024-03-29T11:19:32.167 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.189+0000 7f72e1900640 1 -- start start 2024-03-29T11:19:32.167 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.189+0000 7f72e1900640 10 monclient: init 2024-03-29T11:19:32.170 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.190+0000 7f72e1900640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:32.170 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.190+0000 7f72e1900640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:32.170 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.190+0000 7f72e1900640 10 monclient(hunting): picked mon.noname-b con 0x7f72dc0b0500 addr v1:172.21.15.18:6790/0 2024-03-29T11:19:32.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.190+0000 7f72e1900640 10 monclient(hunting): picked mon.noname-c con 0x7f72dc0b08e0 addr v1:172.21.15.70:6789/0 2024-03-29T11:19:32.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.190+0000 7f72daffd640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f72dc0b0500 0x7f72dc13ff10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:52872/0 (socket says 172.21.15.18:52872) 2024-03-29T11:19:32.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.193+0000 7f72daffd640 1 -- 172.21.15.18:0/1430825475 learned_addr learned my addr 172.21.15.18:0/1430825475 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:19:32.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.193+0000 7f72e1900640 10 monclient(hunting): picked mon.noname-a con 0x7f72dc0aeca0 addr v1:172.21.15.18:6789/0 2024-03-29T11:19:32.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.193+0000 7f72e1900640 1 -- 172.21.15.18:0/1430825475 --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72dc140620 con 0x7f72dc0aeca0 2024-03-29T11:19:32.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.193+0000 7f72e1900640 1 -- 172.21.15.18:0/1430825475 --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72dc13daf0 con 0x7f72dc0b08e0 2024-03-29T11:19:32.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.193+0000 7f72e1900640 1 -- 172.21.15.18:0/1430825475 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72dc0ae180 con 0x7f72dc0b0500 2024-03-29T11:19:32.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.193+0000 7f72e1900640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:32.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.193+0000 7f72e1900640 10 monclient(hunting): authenticate will time out at 1398.465366s 2024-03-29T11:19:32.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.193+0000 7f72d9ffb640 1 -- 172.21.15.18:0/1430825475 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2950493325 0 0) 0x7f72dc0ae180 con 0x7f72dc0b0500 2024-03-29T11:19:32.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.193+0000 7f72d9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:32.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.193+0000 7f72d9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:32.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.193+0000 7f72d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:32.171 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.193+0000 7f72d9ffb640 10 monclient(hunting): my global_id is 4670 2024-03-29T11:19:32.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.193+0000 7f72d9ffb640 1 -- 172.21.15.18:0/1430825475 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72c4003520 con 0x7f72dc0b0500 2024-03-29T11:19:32.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.193+0000 7f72d9ffb640 1 -- 172.21.15.18:0/1430825475 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 171460658 0 0) 0x7f72dc13daf0 con 0x7f72dc0b08e0 2024-03-29T11:19:32.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.193+0000 7f72d9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:32.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.193+0000 7f72d9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:32.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.193+0000 7f72d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:32.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.193+0000 7f72d9ffb640 10 monclient(hunting): my global_id is 4525 2024-03-29T11:19:32.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.193+0000 7f72d9ffb640 1 -- 172.21.15.18:0/1430825475 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72dc0ae180 con 0x7f72dc0b08e0 2024-03-29T11:19:32.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.194+0000 7f72d9ffb640 1 -- 172.21.15.18:0/1430825475 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 75258157 0 0) 0x7f72dc140620 con 0x7f72dc0aeca0 2024-03-29T11:19:32.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.194+0000 7f72d9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:32.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.194+0000 7f72d9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:32.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.194+0000 7f72d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:32.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.194+0000 7f72d9ffb640 10 monclient(hunting): my global_id is 4578 2024-03-29T11:19:32.172 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.194+0000 7f72d9ffb640 1 -- 172.21.15.18:0/1430825475 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72dc13daf0 con 0x7f72dc0aeca0 2024-03-29T11:19:32.175 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.194+0000 7f72d9ffb640 1 -- 172.21.15.18:0/1430825475 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2010598007 0 0) 0x7f72dc0ae180 con 0x7f72dc0b08e0 2024-03-29T11:19:32.175 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.194+0000 7f72d9ffb640 1 -- 172.21.15.18:0/1430825475 >> v1:172.21.15.18:6790/0 conn(0x7f72dc0b0500 legacy=0x7f72dc13ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:32.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.194+0000 7f72d9ffb640 1 -- 172.21.15.18:0/1430825475 >> v1:172.21.15.18:6789/0 conn(0x7f72dc0aeca0 legacy=0x7f72dc14b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:32.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.194+0000 7f72d9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:32.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.194+0000 7f72d9ffb640 1 monclient: found mon.noname-c 2024-03-29T11:19:32.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.194+0000 7f72d9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:32.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.194+0000 7f72d9ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.70:6789/0 2024-03-29T11:19:32.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.194+0000 7f72d9ffb640 1 -- 172.21.15.18:0/1430825475 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72dc0accf0 con 0x7f72dc0b08e0 2024-03-29T11:19:32.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.194+0000 7f72d9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:32.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.194+0000 7f72d9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:32.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.194+0000 7f72d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:32.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.194+0000 7f72d9ffb640 1 -- 172.21.15.18:0/1430825475 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f72d0001670 con 0x7f72dc0b08e0 2024-03-29T11:19:32.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.194+0000 7f72d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:32.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.194+0000 7f72d9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:32.176 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.194+0000 7f72d9ffb640 10 monclient: dump: 2024-03-29T11:19:32.177 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:19:32.177 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:32.177 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:32.177 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:32.177 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:32.177 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:32.177 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:19:32.177 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:19:32.177 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:19:32.177 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:32.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.197+0000 7f72e1900640 5 monclient: authenticate success, global_id 4525 2024-03-29T11:19:32.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.197+0000 7f72e1900640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:32.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.197+0000 7f72d9ffb640 1 -- 172.21.15.18:0/1430825475 <== mon.1 v1:172.21.15.70:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f72d00017d0 con 0x7f72dc0b08e0 2024-03-29T11:19:32.177 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.197+0000 7f72d9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:32.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.197+0000 7f72d9ffb640 1 -- 172.21.15.18:0/1430825475 <== mon.1 v1:172.21.15.70:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f72d0001c20 con 0x7f72dc0b08e0 2024-03-29T11:19:32.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.197+0000 7f72d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:32.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.197+0000 7f72d9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:32.178 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.197+0000 7f72d9ffb640 10 monclient: dump: 2024-03-29T11:19:32.178 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:19:32.178 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:32.178 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:32.178 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:32.178 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:32.178 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:32.178 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:19:32.178 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:19:32.178 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:19:32.178 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:32.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.197+0000 7f72e1900640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:32.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.197+0000 7f72e1900640 4 set_mon_vals no callback set 2024-03-29T11:19:32.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.198+0000 7f72e1900640 10 monclient: shutdown 2024-03-29T11:19:32.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.198+0000 7f72e1900640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:32.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.198+0000 7f72e1900640 1 -- 172.21.15.18:0/1430825475 >> v1:172.21.15.70:6789/0 conn(0x7f72dc0b08e0 legacy=0x7f72dc13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:32.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.199+0000 7f72e1900640 1 -- 172.21.15.18:0/1430825475 shutdown_connections 2024-03-29T11:19:32.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.199+0000 7f72e1900640 1 -- 172.21.15.18:0/1430825475 >> 172.21.15.18:0/1430825475 conn(0x7f72dc06b660 msgr2=0x7f72dc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:32.179 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.199+0000 7f72e1900640 1 -- 172.21.15.18:0/1430825475 shutdown_connections 2024-03-29T11:19:32.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.199+0000 7f72e1900640 1 -- 172.21.15.18:0/1430825475 wait complete. 2024-03-29T11:19:32.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.200+0000 7f72e1900640 10 monclient: build_initial_monmap 2024-03-29T11:19:32.181 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.200+0000 7f72e1900640 10 monclient: monmap: 2024-03-29T11:19:32.181 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:19:32.181 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:32.181 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:19:32.181 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:19:32.181 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:32.181 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:32.181 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:19:32.181 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:19:32.181 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:19:32.181 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:32.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.200+0000 7f72e1900640 1 librados: starting msgr at 2024-03-29T11:19:32.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.200+0000 7f72e1900640 1 librados: starting objecter 2024-03-29T11:19:32.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.200+0000 7f72e1900640 1 Processor -- start 2024-03-29T11:19:32.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.201+0000 7f72e1900640 1 -- start start 2024-03-29T11:19:32.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.201+0000 7f72e1900640 1 librados: setting wanted keys 2024-03-29T11:19:32.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.201+0000 7f72e1900640 1 librados: calling monclient init 2024-03-29T11:19:32.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.201+0000 7f72e1900640 10 monclient: init 2024-03-29T11:19:32.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.201+0000 7f72e1900640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:32.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.201+0000 7f72e1900640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:32.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.202+0000 7f72e1900640 10 monclient(hunting): picked mon.noname-a con 0x7f72dc0aeca0 addr v1:172.21.15.18:6789/0 2024-03-29T11:19:32.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.202+0000 7f72e1900640 10 monclient(hunting): picked mon.noname-c con 0x7f72dc0b0500 addr v1:172.21.15.18:6790/0 2024-03-29T11:19:32.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.202+0000 7f72e1900640 10 monclient(hunting): picked mon.noname-b con 0x7f72dc0701c0 addr v1:172.21.15.70:6789/0 2024-03-29T11:19:32.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.202+0000 7f72e1900640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72dc06f520 con 0x7f72dc0aeca0 2024-03-29T11:19:32.182 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.202+0000 7f72e1900640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72dc06ea80 con 0x7f72dc0701c0 2024-03-29T11:19:32.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.202+0000 7f72e1900640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72dc06f180 con 0x7f72dc0b0500 2024-03-29T11:19:32.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.202+0000 7f72e1900640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:32.183 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.202+0000 7f72e1900640 10 monclient(hunting): authenticate will time out at 1398.474504s 2024-03-29T11:19:32.188 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.205+0000 7f72da7fc640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f72dc0b0500 0x7f72dc06f9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:58114/0 (socket says 172.21.15.18:58114) 2024-03-29T11:19:32.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.205+0000 7f72da7fc640 1 -- 172.21.15.18:0/995921209 learned_addr learned my addr 172.21.15.18:0/995921209 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:19:32.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.205+0000 7f72c3fff640 10 client.?.objecter ms_handle_connect 0x7f72dc0b0500 2024-03-29T11:19:32.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.205+0000 7f72c3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:32.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.205+0000 7f72c3fff640 10 client.?.objecter ms_handle_connect 0x7f72dc0701c0 2024-03-29T11:19:32.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.205+0000 7f72c3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:32.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.206+0000 7f72c3fff640 1 -- 172.21.15.18:0/995921209 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3096635687 0 0) 0x7f72dc06f180 con 0x7f72dc0b0500 2024-03-29T11:19:32.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.206+0000 7f72c3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:19:32.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.206+0000 7f72c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:32.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.206+0000 7f72c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:32.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.206+0000 7f72c3fff640 10 monclient(hunting): my global_id is 4673 2024-03-29T11:19:32.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.206+0000 7f72c3fff640 1 -- 172.21.15.18:0/995921209 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72c8004eb0 con 0x7f72dc0b0500 2024-03-29T11:19:32.189 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.206+0000 7f72c3fff640 1 -- 172.21.15.18:0/995921209 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3452605922 0 0) 0x7f72c8004eb0 con 0x7f72dc0b0500 2024-03-29T11:19:32.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.206+0000 7f72c3fff640 1 -- 172.21.15.18:0/995921209 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72c8005300 con 0x7f72dc0b0500 2024-03-29T11:19:32.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.206+0000 7f72c3fff640 1 -- 172.21.15.18:0/995921209 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f72d0001c20 con 0x7f72dc0b0500 2024-03-29T11:19:32.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.206+0000 7f72c3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:32.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.206+0000 7f72c3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:32.190 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.206+0000 7f72c3fff640 10 monclient(hunting): dump: 2024-03-29T11:19:32.190 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:19:32.190 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:32.190 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:32.190 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:32.190 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:32.190 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:32.190 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:19:32.190 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:19:32.191 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:19:32.191 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:32.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.206+0000 7f72c3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:32.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.206+0000 7f72c3fff640 1 -- 172.21.15.18:0/995921209 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1615194425 0 0) 0x7f72c8005300 con 0x7f72dc0b0500 2024-03-29T11:19:32.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.206+0000 7f72c3fff640 1 -- 172.21.15.18:0/995921209 >> v1:172.21.15.70:6789/0 conn(0x7f72dc0701c0 legacy=0x7f72dc1e6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:32.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.206+0000 7f72c3fff640 1 -- 172.21.15.18:0/995921209 >> v1:172.21.15.18:6789/0 conn(0x7f72dc0aeca0 legacy=0x7f72dc1c6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:32.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.206+0000 7f72c3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:19:32.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.206+0000 7f72c3fff640 1 monclient: found mon.c 2024-03-29T11:19:32.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.206+0000 7f72c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:32.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.206+0000 7f72c3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:19:32.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.206+0000 7f72c3fff640 1 -- 172.21.15.18:0/995921209 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72dc13a620 con 0x7f72dc0b0500 2024-03-29T11:19:32.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.206+0000 7f72c3fff640 10 monclient: _finish_auth 0 2024-03-29T11:19:32.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.206+0000 7f72c3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:19:32.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.206+0000 7f72c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:32.191 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.206+0000 7f72c3fff640 1 -- 172.21.15.18:0/995921209 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f72d00038a0 con 0x7f72dc0b0500 2024-03-29T11:19:32.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.206+0000 7f72c3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:32.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.206+0000 7f72d97fa640 4 set_mon_vals no callback set 2024-03-29T11:19:32.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.207+0000 7f72c3fff640 1 -- 172.21.15.18:0/995921209 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f72d0001c20 con 0x7f72dc0b0500 2024-03-29T11:19:32.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.207+0000 7f72c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:32.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.207+0000 7f72c3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:32.192 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.207+0000 7f72c3fff640 10 monclient: dump: 2024-03-29T11:19:32.192 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:19:32.192 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:32.192 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:32.192 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:32.192 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:32.192 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:32.192 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:19:32.192 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:19:32.192 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:19:32.192 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:32.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.208+0000 7f72e1900640 5 monclient: authenticate success, global_id 4673 2024-03-29T11:19:32.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.208+0000 7f72e1900640 10 monclient: _renew_subs 2024-03-29T11:19:32.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.208+0000 7f72e1900640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:19:32.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.208+0000 7f72e1900640 1 -- 172.21.15.18:0/995921209 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72dc139aa0 con 0x7f72dc0b0500 2024-03-29T11:19:32.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.208+0000 7f72e1900640 10 client.4673.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:32.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.208+0000 7f72e1900640 10 monclient: _renew_subs 2024-03-29T11:19:32.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.208+0000 7f72e1900640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:19:32.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.208+0000 7f72e1900640 1 -- 172.21.15.18:0/995921209 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f72dc139e40 con 0x7f72dc0b0500 2024-03-29T11:19:32.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.208+0000 7f72e1900640 1 librados: init done 2024-03-29T11:19:32.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.210+0000 7f72e1900640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:32.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.210+0000 7f72e1900640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:32.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.210+0000 7f72e1900640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:19:32.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.210+0000 7f72e1900640 1 -- 172.21.15.18:0/995921209 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72dc0ad0c0 con 0x7f72dc0b0500 2024-03-29T11:19:32.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.211+0000 7f72c3fff640 1 -- 172.21.15.18:0/995921209 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f72d0003a90 con 0x7f72dc0b0500 2024-03-29T11:19:32.193 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.211+0000 7f72c3fff640 1 -- 172.21.15.18:0/995921209 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3599602952 0 0) 0x7f72d0054430 con 0x7f72dc0b0500 2024-03-29T11:19:32.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.211+0000 7f72c3fff640 10 client.4673.objecter ms_dispatch 0x7f72dc000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:19:32.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.211+0000 7f72c3fff640 3 client.4673.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:19:32.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.211+0000 7f72c3fff640 3 client.4673.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:19:32.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.211+0000 7f72c3fff640 20 client.4673.objecter dump_active .. 0 homeless 2024-03-29T11:19:32.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.212+0000 7f72daffd640 10 monclient: get_auth_request con 0x7f72c806e2c0 auth_method 0 2024-03-29T11:19:32.194 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.214+0000 7f72c3fff640 10 client.4673.objecter ms_handle_connect 0x7f72c806e2c0 2024-03-29T11:19:32.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.219+0000 7f72c3fff640 1 -- 172.21.15.18:0/995921209 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f72d00506d0 con 0x7f72dc0b0500 2024-03-29T11:19:32.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.219+0000 7f72c3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:32.196 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.219+0000 7f72c3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:32.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.551+0000 7f72e1900640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:32.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.551+0000 7f72e1900640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:32.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.551+0000 7f72e1900640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:19:32.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.551+0000 7f72e1900640 1 -- 172.21.15.18:0/995921209 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72dc0ad0c0 con 0x7f72dc0b0500 2024-03-29T11:19:32.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.552+0000 7f72c3fff640 1 -- 172.21.15.18:0/995921209 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2084344819) 0x7f72d00506d0 con 0x7f72dc0b0500 2024-03-29T11:19:32.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.552+0000 7f72c3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:32.529 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.552+0000 7f72c3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:32.529 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:19:32.529 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":11,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:19:08.996105+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:08.996105+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:32.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.555+0000 7f72e1900640 10 librados: watch_flush enter 2024-03-29T11:19:32.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.555+0000 7f72e1900640 10 librados: watch_flush exit 2024-03-29T11:19:32.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.555+0000 7f72e1900640 20 client.4673.objecter shutdown clearing up homeless session... 2024-03-29T11:19:32.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.555+0000 7f72e1900640 10 client.4673.objecter successfully canceled tick 2024-03-29T11:19:32.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.555+0000 7f72e1900640 1 -- 172.21.15.18:0/995921209 >> v1:172.21.15.70:6816/3101871562 conn(0x7f72c806e2c0 legacy=0x7f72c8070780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:32.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.555+0000 7f72e1900640 10 monclient: shutdown 2024-03-29T11:19:32.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.555+0000 7f72e1900640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:32.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.555+0000 7f72e1900640 1 -- 172.21.15.18:0/995921209 >> v1:172.21.15.18:6790/0 conn(0x7f72dc0b0500 legacy=0x7f72dc06f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:32.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.556+0000 7f72e1900640 1 -- 172.21.15.18:0/995921209 shutdown_connections 2024-03-29T11:19:32.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.556+0000 7f72e1900640 1 -- 172.21.15.18:0/995921209 >> 172.21.15.18:0/995921209 conn(0x7f72dc06b660 msgr2=0x7f72dc06ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:32.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.556+0000 7f72e1900640 1 -- 172.21.15.18:0/995921209 shutdown_connections 2024-03-29T11:19:32.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.556+0000 7f72e1900640 1 -- 172.21.15.18:0/995921209 wait complete. 2024-03-29T11:19:32.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:32.556+0000 7f72e1900640 1 librados: shutdown 2024-03-29T11:19:32.547 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:39.354 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:39.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.549+0000 7f14f52d3640 10 monclient: get_monmap_and_config 2024-03-29T11:19:39.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.549+0000 7f14f52d3640 10 monclient: build_initial_monmap 2024-03-29T11:19:39.526 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.549+0000 7f14f52d3640 10 monclient: monmap: 2024-03-29T11:19:39.526 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:19:39.526 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:39.526 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:19:39.550029+0000 2024-03-29T11:19:39.526 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:19:39.550029+0000 2024-03-29T11:19:39.526 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:39.526 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:39.526 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:19:39.527 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:19:39.527 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:19:39.527 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:39.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.550+0000 7f14f52d3640 1 Processor -- start 2024-03-29T11:19:39.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.550+0000 7f14f52d3640 1 -- start start 2024-03-29T11:19:39.527 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.550+0000 7f14f52d3640 10 monclient: init 2024-03-29T11:19:39.528 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.551+0000 7f14f52d3640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:39.531 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.551+0000 7f14f52d3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:19:39.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.555+0000 7f14f52d3640 10 monclient(hunting): picked mon.noname-a con 0x7f14f00b0770 addr v1:172.21.15.18:6789/0 2024-03-29T11:19:39.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.556+0000 7f14f52d3640 10 monclient(hunting): picked mon.noname-c con 0x7f14f014a700 addr v1:172.21.15.70:6789/0 2024-03-29T11:19:39.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.556+0000 7f14f52d3640 10 monclient(hunting): picked mon.noname-b con 0x7f14f00ac5a0 addr v1:172.21.15.18:6790/0 2024-03-29T11:19:39.533 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.556+0000 7f14f52d3640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14f00aba70 con 0x7f14f00b0770 2024-03-29T11:19:39.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.556+0000 7f14f52d3640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14f00aecb0 con 0x7f14f014a700 2024-03-29T11:19:39.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.556+0000 7f14f52d3640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14f00af150 con 0x7f14f00ac5a0 2024-03-29T11:19:39.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.556+0000 7f14f52d3640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:39.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.556+0000 7f14f52d3640 10 monclient(hunting): authenticate will time out at 1405.828013s 2024-03-29T11:19:39.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.556+0000 7f14ef7fe640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f14f00ac5a0 0x7f14f0151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:41260/0 (socket says 172.21.15.18:41260) 2024-03-29T11:19:39.534 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.556+0000 7f14ef7fe640 1 -- 172.21.15.18:0/2392347024 learned_addr learned my addr 172.21.15.18:0/2392347024 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:19:39.536 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.560+0000 7f14edffb640 1 -- 172.21.15.18:0/2392347024 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4067685213 0 0) 0x7f14f00aecb0 con 0x7f14f014a700 2024-03-29T11:19:39.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.560+0000 7f14edffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:39.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.560+0000 7f14edffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:39.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.560+0000 7f14edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:39.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.560+0000 7f14edffb640 10 monclient(hunting): my global_id is 4528 2024-03-29T11:19:39.537 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.560+0000 7f14edffb640 1 -- 172.21.15.18:0/2392347024 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14d8003520 con 0x7f14f014a700 2024-03-29T11:19:39.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.560+0000 7f14edffb640 1 -- 172.21.15.18:0/2392347024 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544361515 0 0) 0x7f14f00aba70 con 0x7f14f00b0770 2024-03-29T11:19:39.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.560+0000 7f14edffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:39.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.560+0000 7f14edffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:39.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.560+0000 7f14edffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:39.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.560+0000 7f14edffb640 10 monclient(hunting): my global_id is 4581 2024-03-29T11:19:39.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.560+0000 7f14edffb640 1 -- 172.21.15.18:0/2392347024 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14f00aecb0 con 0x7f14f00b0770 2024-03-29T11:19:39.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.560+0000 7f14edffb640 1 -- 172.21.15.18:0/2392347024 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3790614221 0 0) 0x7f14d8003520 con 0x7f14f014a700 2024-03-29T11:19:39.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.560+0000 7f14edffb640 1 -- 172.21.15.18:0/2392347024 >> v1:172.21.15.18:6790/0 conn(0x7f14f00ac5a0 legacy=0x7f14f0151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:39.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.560+0000 7f14edffb640 1 -- 172.21.15.18:0/2392347024 >> v1:172.21.15.18:6789/0 conn(0x7f14f00b0770 legacy=0x7f14f0149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:39.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.560+0000 7f14edffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:39.538 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.560+0000 7f14edffb640 1 monclient: found mon.noname-c 2024-03-29T11:19:39.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.560+0000 7f14edffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:39.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.560+0000 7f14edffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.70:6789/0 2024-03-29T11:19:39.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.560+0000 7f14edffb640 1 -- 172.21.15.18:0/2392347024 --> v1:172.21.15.70:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f00ac9d0 con 0x7f14f014a700 2024-03-29T11:19:39.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.560+0000 7f14edffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:39.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.560+0000 7f14edffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:39.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.560+0000 7f14edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:39.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.560+0000 7f14edffb640 1 -- 172.21.15.18:0/2392347024 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f14e8001a30 con 0x7f14f014a700 2024-03-29T11:19:39.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.560+0000 7f14edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:39.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.560+0000 7f14edffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:39.539 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.560+0000 7f14edffb640 10 monclient: dump: 2024-03-29T11:19:39.539 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:19:39.539 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:39.539 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:39.539 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:39.540 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:39.540 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:39.540 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:19:39.540 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:19:39.540 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:19:39.540 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:39.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.561+0000 7f14edffb640 1 -- 172.21.15.18:0/2392347024 <== mon.1 v1:172.21.15.70:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14e8001b90 con 0x7f14f014a700 2024-03-29T11:19:39.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.561+0000 7f14edffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:39.540 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.561+0000 7f14edffb640 1 -- 172.21.15.18:0/2392347024 <== mon.1 v1:172.21.15.70:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f14e8001fa0 con 0x7f14f014a700 2024-03-29T11:19:39.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.561+0000 7f14edffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:39.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.561+0000 7f14edffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:39.542 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.561+0000 7f14edffb640 10 monclient: dump: 2024-03-29T11:19:39.542 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:19:39.542 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:39.542 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:39.542 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:39.543 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:39.543 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:39.543 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:19:39.543 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:19:39.543 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:19:39.543 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:39.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.562+0000 7f14f52d3640 5 monclient: authenticate success, global_id 4528 2024-03-29T11:19:39.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.562+0000 7f14f52d3640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:39.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.562+0000 7f14f52d3640 4 set_mon_vals no callback set 2024-03-29T11:19:39.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.563+0000 7f14f52d3640 10 monclient: shutdown 2024-03-29T11:19:39.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.563+0000 7f14f52d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:39.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.563+0000 7f14f52d3640 1 -- 172.21.15.18:0/2392347024 >> v1:172.21.15.70:6789/0 conn(0x7f14f014a700 legacy=0x7f14f00abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:39.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.565+0000 7f14f52d3640 1 -- 172.21.15.18:0/2392347024 shutdown_connections 2024-03-29T11:19:39.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.565+0000 7f14f52d3640 1 -- 172.21.15.18:0/2392347024 >> 172.21.15.18:0/2392347024 conn(0x7f14f006b660 msgr2=0x7f14f006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:39.543 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.565+0000 7f14f52d3640 1 -- 172.21.15.18:0/2392347024 shutdown_connections 2024-03-29T11:19:39.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.565+0000 7f14f52d3640 1 -- 172.21.15.18:0/2392347024 wait complete. 2024-03-29T11:19:39.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.566+0000 7f14f52d3640 10 monclient: build_initial_monmap 2024-03-29T11:19:39.544 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.566+0000 7f14f52d3640 10 monclient: monmap: 2024-03-29T11:19:39.544 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:19:39.544 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:39.544 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:19:39.544 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:19:39.544 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:39.544 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:39.544 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:19:39.544 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:19:39.544 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:19:39.545 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:39.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.567+0000 7f14f52d3640 1 librados: starting msgr at 2024-03-29T11:19:39.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.567+0000 7f14f52d3640 1 librados: starting objecter 2024-03-29T11:19:39.545 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.567+0000 7f14f52d3640 1 Processor -- start 2024-03-29T11:19:39.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.567+0000 7f14f52d3640 1 -- start start 2024-03-29T11:19:39.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.567+0000 7f14f52d3640 1 librados: setting wanted keys 2024-03-29T11:19:39.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.567+0000 7f14f52d3640 1 librados: calling monclient init 2024-03-29T11:19:39.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.567+0000 7f14f52d3640 10 monclient: init 2024-03-29T11:19:39.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.568+0000 7f14f52d3640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:39.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.568+0000 7f14f52d3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:39.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.568+0000 7f14f52d3640 10 monclient(hunting): picked mon.noname-c con 0x7f14f00ac5a0 addr v1:172.21.15.18:6790/0 2024-03-29T11:19:39.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.568+0000 7f14f52d3640 10 monclient(hunting): picked mon.noname-b con 0x7f14f00b0770 addr v1:172.21.15.70:6789/0 2024-03-29T11:19:39.546 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.568+0000 7f14eeffd640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f14f00ac5a0 0x7f14f01e4080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:42048/0 (socket says 172.21.15.18:42048) 2024-03-29T11:19:39.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.571+0000 7f14f52d3640 10 monclient(hunting): picked mon.noname-a con 0x7f14f014a700 addr v1:172.21.15.18:6789/0 2024-03-29T11:19:39.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.571+0000 7f14f52d3640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14f0122c40 con 0x7f14f014a700 2024-03-29T11:19:39.549 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.571+0000 7f14f52d3640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14f0121460 con 0x7f14f00b0770 2024-03-29T11:19:39.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.571+0000 7f14f52d3640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14f0121080 con 0x7f14f00ac5a0 2024-03-29T11:19:39.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.571+0000 7f14f52d3640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:39.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.571+0000 7f14f52d3640 10 monclient(hunting): authenticate will time out at 1405.843698s 2024-03-29T11:19:39.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.571+0000 7f14eeffd640 1 -- 172.21.15.18:0/2921948395 learned_addr learned my addr 172.21.15.18:0/2921948395 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:19:39.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.572+0000 7f14d77fe640 10 client.?.objecter ms_handle_connect 0x7f14f00ac5a0 2024-03-29T11:19:39.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.572+0000 7f14d77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:39.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.572+0000 7f14d77fe640 1 -- 172.21.15.18:0/2921948395 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1260128031 0 0) 0x7f14f0121080 con 0x7f14f00ac5a0 2024-03-29T11:19:39.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.572+0000 7f14d77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:39.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.572+0000 7f14d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:39.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.572+0000 7f14d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:39.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.572+0000 7f14d77fe640 10 monclient(hunting): my global_id is 4679 2024-03-29T11:19:39.550 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.572+0000 7f14d77fe640 1 -- 172.21.15.18:0/2921948395 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14dc004eb0 con 0x7f14f00ac5a0 2024-03-29T11:19:39.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.575+0000 7f14d77fe640 1 -- 172.21.15.18:0/2921948395 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 461006891 0 0) 0x7f14dc004eb0 con 0x7f14f00ac5a0 2024-03-29T11:19:39.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.575+0000 7f14d77fe640 1 -- 172.21.15.18:0/2921948395 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14f0121080 con 0x7f14f00ac5a0 2024-03-29T11:19:39.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.575+0000 7f14d77fe640 1 -- 172.21.15.18:0/2921948395 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f14e4001a10 con 0x7f14f00ac5a0 2024-03-29T11:19:39.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.575+0000 7f14d77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:39.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.575+0000 7f14d77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:39.555 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.575+0000 7f14d77fe640 10 monclient(hunting): dump: 2024-03-29T11:19:39.555 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:19:39.555 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:39.555 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:39.555 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:39.555 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:39.556 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:39.556 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:19:39.556 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:19:39.556 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:19:39.556 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:39.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.575+0000 7f14d77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:39.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.576+0000 7f14d77fe640 10 client.?.objecter ms_handle_connect 0x7f14f00b0770 2024-03-29T11:19:39.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.576+0000 7f14d77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:39.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.576+0000 7f14d77fe640 1 -- 172.21.15.18:0/2921948395 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3744634083 0 0) 0x7f14f0121460 con 0x7f14f00b0770 2024-03-29T11:19:39.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.576+0000 7f14d77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:39.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.576+0000 7f14d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:39.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.576+0000 7f14d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:39.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.576+0000 7f14d77fe640 10 monclient(hunting): my global_id is 4531 2024-03-29T11:19:39.556 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.576+0000 7f14d77fe640 1 -- 172.21.15.18:0/2921948395 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14dc004eb0 con 0x7f14f00b0770 2024-03-29T11:19:39.557 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.576+0000 7f14d77fe640 10 client.?.objecter ms_handle_connect 0x7f14f014a700 2024-03-29T11:19:39.557 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.576+0000 7f14d77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:39.557 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.576+0000 7f14d77fe640 1 -- 172.21.15.18:0/2921948395 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 956668309 0 0) 0x7f14dc004eb0 con 0x7f14f00b0770 2024-03-29T11:19:39.557 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.576+0000 7f14d77fe640 1 -- 172.21.15.18:0/2921948395 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14f0121460 con 0x7f14f00b0770 2024-03-29T11:19:39.557 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.577+0000 7f14d77fe640 1 -- 172.21.15.18:0/2921948395 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f14e8001ea0 con 0x7f14f00b0770 2024-03-29T11:19:39.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.577+0000 7f14d77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:39.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.577+0000 7f14d77fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:39.559 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.577+0000 7f14d77fe640 10 monclient(hunting): dump: 2024-03-29T11:19:39.559 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:19:39.560 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:39.560 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:39.560 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:39.560 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:39.560 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:39.560 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:19:39.560 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:19:39.560 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:19:39.560 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:39.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.580+0000 7f14d77fe640 1 -- 172.21.15.18:0/2921948395 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1559899558 0 0) 0x7f14f0122c40 con 0x7f14f014a700 2024-03-29T11:19:39.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.580+0000 7f14d77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:39.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.580+0000 7f14d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:39.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.580+0000 7f14d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:39.560 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.580+0000 7f14d77fe640 10 monclient(hunting): my global_id is 4584 2024-03-29T11:19:39.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.580+0000 7f14d77fe640 1 -- 172.21.15.18:0/2921948395 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14dc004eb0 con 0x7f14f014a700 2024-03-29T11:19:39.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.580+0000 7f14d77fe640 1 -- 172.21.15.18:0/2921948395 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2388941009 0 0) 0x7f14f0121080 con 0x7f14f00ac5a0 2024-03-29T11:19:39.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.581+0000 7f14d77fe640 1 -- 172.21.15.18:0/2921948395 >> v1:172.21.15.70:6789/0 conn(0x7f14f00b0770 legacy=0x7f14f01dac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:39.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.581+0000 7f14d77fe640 1 -- 172.21.15.18:0/2921948395 >> v1:172.21.15.18:6789/0 conn(0x7f14f014a700 legacy=0x7f14f01ef8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:39.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.581+0000 7f14d77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:39.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.581+0000 7f14d77fe640 1 monclient: found mon.c 2024-03-29T11:19:39.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.581+0000 7f14d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:39.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.581+0000 7f14d77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:19:39.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.581+0000 7f14d77fe640 1 -- 172.21.15.18:0/2921948395 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f0121210 con 0x7f14f00ac5a0 2024-03-29T11:19:39.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.581+0000 7f14d77fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:39.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.581+0000 7f14d77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:39.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.581+0000 7f14d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:39.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.581+0000 7f14f52d3640 5 monclient: authenticate success, global_id 4679 2024-03-29T11:19:39.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.581+0000 7f14f52d3640 10 monclient: _renew_subs 2024-03-29T11:19:39.561 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.581+0000 7f14f52d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:19:39.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.581+0000 7f14f52d3640 1 -- 172.21.15.18:0/2921948395 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14f0121a40 con 0x7f14f00ac5a0 2024-03-29T11:19:39.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.581+0000 7f14f52d3640 10 client.4679.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:39.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.581+0000 7f14f52d3640 10 monclient: _renew_subs 2024-03-29T11:19:39.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.581+0000 7f14f52d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:19:39.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.581+0000 7f14f52d3640 1 -- 172.21.15.18:0/2921948395 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f14f01e4970 con 0x7f14f00ac5a0 2024-03-29T11:19:39.562 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.581+0000 7f14f52d3640 1 librados: init done 2024-03-29T11:19:39.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.583+0000 7f14f52d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:39.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.584+0000 7f14f52d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:39.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.584+0000 7f14f52d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:19:39.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.584+0000 7f14f52d3640 1 -- 172.21.15.18:0/2921948395 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14f00ab560 con 0x7f14f00ac5a0 2024-03-29T11:19:39.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.585+0000 7f14d77fe640 1 -- 172.21.15.18:0/2921948395 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14e4001b70 con 0x7f14f00ac5a0 2024-03-29T11:19:39.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.585+0000 7f14d77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:39.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.585+0000 7f14d77fe640 1 -- 172.21.15.18:0/2921948395 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f14e4003a60 con 0x7f14f00ac5a0 2024-03-29T11:19:39.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.585+0000 7f14d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:39.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.585+0000 7f14d77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:39.564 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.585+0000 7f14d77fe640 10 monclient: dump: 2024-03-29T11:19:39.564 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:19:39.564 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:39.565 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:39.565 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:39.565 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:39.565 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:39.565 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:19:39.565 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:19:39.565 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:19:39.565 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:39.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.585+0000 7f14d77fe640 1 -- 172.21.15.18:0/2921948395 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f14e4003c80 con 0x7f14f00ac5a0 2024-03-29T11:19:39.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.585+0000 7f14ecff9640 4 set_mon_vals no callback set 2024-03-29T11:19:39.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.586+0000 7f14d77fe640 1 -- 172.21.15.18:0/2921948395 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3599602952 0 0) 0x7f14e4084b40 con 0x7f14f00ac5a0 2024-03-29T11:19:39.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.587+0000 7f14ee7fc640 10 monclient: get_auth_request con 0x7f14dc06df70 auth_method 0 2024-03-29T11:19:39.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.587+0000 7f14d77fe640 10 client.4679.objecter ms_dispatch 0x7f14f0000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:19:39.565 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.587+0000 7f14d77fe640 3 client.4679.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:19:39.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.587+0000 7f14d77fe640 3 client.4679.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:19:39.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.587+0000 7f14d77fe640 20 client.4679.objecter dump_active .. 0 homeless 2024-03-29T11:19:39.566 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.587+0000 7f14d77fe640 10 client.4679.objecter ms_handle_connect 0x7f14dc06df70 2024-03-29T11:19:39.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.596+0000 7f14d77fe640 1 -- 172.21.15.18:0/2921948395 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f14e4050420 con 0x7f14f00ac5a0 2024-03-29T11:19:39.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.596+0000 7f14d77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:39.573 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.596+0000 7f14d77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:39.938 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.961+0000 7f14f52d3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:39.939 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.961+0000 7f14f52d3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:39.939 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.961+0000 7f14f52d3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:19:39.939 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.961+0000 7f14f52d3640 1 -- 172.21.15.18:0/2921948395 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14f01215f0 con 0x7f14f00ac5a0 2024-03-29T11:19:39.942 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.965+0000 7f14d77fe640 1 -- 172.21.15.18:0/2921948395 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2084344819) 0x7f14e4052350 con 0x7f14f00ac5a0 2024-03-29T11:19:39.942 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.965+0000 7f14d77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:39.942 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.965+0000 7f14d77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:39.943 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:19:39.943 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":11,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:19:08.996105+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:08.996105+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:39.946 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.969+0000 7f14f52d3640 10 librados: watch_flush enter 2024-03-29T11:19:39.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.970+0000 7f14f52d3640 10 librados: watch_flush exit 2024-03-29T11:19:39.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.970+0000 7f14f52d3640 20 client.4679.objecter shutdown clearing up homeless session... 2024-03-29T11:19:39.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.970+0000 7f14f52d3640 10 client.4679.objecter successfully canceled tick 2024-03-29T11:19:39.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.970+0000 7f14f52d3640 1 -- 172.21.15.18:0/2921948395 >> v1:172.21.15.70:6816/3101871562 conn(0x7f14dc06df70 legacy=0x7f14dc070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:39.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.970+0000 7f14f52d3640 10 monclient: shutdown 2024-03-29T11:19:39.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.970+0000 7f14f52d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:39.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.970+0000 7f14f52d3640 1 -- 172.21.15.18:0/2921948395 >> v1:172.21.15.18:6790/0 conn(0x7f14f00ac5a0 legacy=0x7f14f01e4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:39.947 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.970+0000 7f14f52d3640 1 -- 172.21.15.18:0/2921948395 shutdown_connections 2024-03-29T11:19:39.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.970+0000 7f14f52d3640 1 -- 172.21.15.18:0/2921948395 >> 172.21.15.18:0/2921948395 conn(0x7f14f006b660 msgr2=0x7f14f00b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:39.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.971+0000 7f14f52d3640 1 -- 172.21.15.18:0/2921948395 shutdown_connections 2024-03-29T11:19:39.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.971+0000 7f14f52d3640 1 -- 172.21.15.18:0/2921948395 wait complete. 2024-03-29T11:19:39.948 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:39.971+0000 7f14f52d3640 1 librados: shutdown 2024-03-29T11:19:39.964 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:46.770 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:46.940 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.962+0000 7f7a43fff640 10 monclient: get_monmap_and_config 2024-03-29T11:19:46.940 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.962+0000 7f7a43fff640 10 monclient: build_initial_monmap 2024-03-29T11:19:46.940 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.962+0000 7f7a43fff640 10 monclient: monmap: 2024-03-29T11:19:46.940 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:19:46.940 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:46.940 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:19:46.964278+0000 2024-03-29T11:19:46.940 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:19:46.964278+0000 2024-03-29T11:19:46.940 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:46.940 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:46.940 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:19:46.940 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:19:46.941 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:19:46.941 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:46.944 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.963+0000 7f7a43fff640 1 Processor -- start 2024-03-29T11:19:46.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.963+0000 7f7a43fff640 1 -- start start 2024-03-29T11:19:46.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.963+0000 7f7a43fff640 10 monclient: init 2024-03-29T11:19:46.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.964+0000 7f7a43fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:46.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.964+0000 7f7a43fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:46.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.964+0000 7f7a43fff640 10 monclient(hunting): picked mon.noname-c con 0x7f7a3c0389e0 addr v1:172.21.15.70:6789/0 2024-03-29T11:19:46.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.964+0000 7f7a43fff640 10 monclient(hunting): picked mon.noname-b con 0x7f7a3c038dc0 addr v1:172.21.15.18:6790/0 2024-03-29T11:19:46.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.965+0000 7f7a427fc640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f7a3c038dc0 0x7f7a3c0cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:57386/0 (socket says 172.21.15.18:57386) 2024-03-29T11:19:46.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.966+0000 7f7a43fff640 10 monclient(hunting): picked mon.noname-a con 0x7f7a3c030660 addr v1:172.21.15.18:6789/0 2024-03-29T11:19:46.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.966+0000 7f7a43fff640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a3c0c54c0 con 0x7f7a3c030660 2024-03-29T11:19:46.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.966+0000 7f7a43fff640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a3c036430 con 0x7f7a3c0389e0 2024-03-29T11:19:46.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.966+0000 7f7a43fff640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a3c039840 con 0x7f7a3c038dc0 2024-03-29T11:19:46.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.966+0000 7f7a43fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:46.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.966+0000 7f7a43fff640 10 monclient(hunting): authenticate will time out at 1413.239327s 2024-03-29T11:19:46.945 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.967+0000 7f7a427fc640 1 -- 172.21.15.18:0/708510266 learned_addr learned my addr 172.21.15.18:0/708510266 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:19:46.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.969+0000 7f7a41ffb640 1 -- 172.21.15.18:0/708510266 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2213770314 0 0) 0x7f7a3c036430 con 0x7f7a3c0389e0 2024-03-29T11:19:46.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.969+0000 7f7a41ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:46.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.969+0000 7f7a41ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:46.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.969+0000 7f7a41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:46.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.969+0000 7f7a41ffb640 10 monclient(hunting): my global_id is 4534 2024-03-29T11:19:46.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.969+0000 7f7a41ffb640 1 -- 172.21.15.18:0/708510266 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a2c003520 con 0x7f7a3c0389e0 2024-03-29T11:19:46.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.969+0000 7f7a41ffb640 1 -- 172.21.15.18:0/708510266 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2483463050 0 0) 0x7f7a3c0c54c0 con 0x7f7a3c030660 2024-03-29T11:19:46.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.969+0000 7f7a41ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:19:46.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.969+0000 7f7a41ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:46.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.969+0000 7f7a41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:46.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.969+0000 7f7a41ffb640 10 monclient(hunting): my global_id is 4587 2024-03-29T11:19:46.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.969+0000 7f7a41ffb640 1 -- 172.21.15.18:0/708510266 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a3c036430 con 0x7f7a3c030660 2024-03-29T11:19:46.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.970+0000 7f7a41ffb640 1 -- 172.21.15.18:0/708510266 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3369269246 0 0) 0x7f7a3c036430 con 0x7f7a3c030660 2024-03-29T11:19:46.949 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.970+0000 7f7a41ffb640 1 -- 172.21.15.18:0/708510266 >> v1:172.21.15.18:6790/0 conn(0x7f7a3c038dc0 legacy=0x7f7a3c0cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:46.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.970+0000 7f7a41ffb640 1 -- 172.21.15.18:0/708510266 >> v1:172.21.15.70:6789/0 conn(0x7f7a3c0389e0 legacy=0x7f7a3c033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:46.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.970+0000 7f7a41ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:19:46.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.970+0000 7f7a41ffb640 1 monclient: found mon.noname-a 2024-03-29T11:19:46.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.970+0000 7f7a41ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:46.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.970+0000 7f7a41ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-29T11:19:46.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.970+0000 7f7a41ffb640 1 -- 172.21.15.18:0/708510266 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a3c035690 con 0x7f7a3c030660 2024-03-29T11:19:46.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.970+0000 7f7a41ffb640 10 monclient: _finish_auth 0 2024-03-29T11:19:46.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.970+0000 7f7a41ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:19:46.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.970+0000 7f7a41ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:46.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.970+0000 7f7a41ffb640 1 -- 172.21.15.18:0/708510266 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f7a4404f730 con 0x7f7a3c030660 2024-03-29T11:19:46.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.970+0000 7f7a41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:46.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.970+0000 7f7a41ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:46.950 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.970+0000 7f7a41ffb640 10 monclient: dump: 2024-03-29T11:19:46.951 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:19:46.951 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:46.951 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:46.951 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:46.951 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:46.951 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:46.951 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:19:46.951 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:19:46.951 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:19:46.951 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:46.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.970+0000 7f7a43fff640 5 monclient: authenticate success, global_id 4587 2024-03-29T11:19:46.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.970+0000 7f7a43fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:46.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.971+0000 7f7a41ffb640 1 -- 172.21.15.18:0/708510266 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a440008d0 con 0x7f7a3c030660 2024-03-29T11:19:46.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.971+0000 7f7a41ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:46.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.971+0000 7f7a41ffb640 1 -- 172.21.15.18:0/708510266 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f7a440758b0 con 0x7f7a3c030660 2024-03-29T11:19:46.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.971+0000 7f7a41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:46.951 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.971+0000 7f7a41ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:46.952 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.971+0000 7f7a41ffb640 10 monclient: dump: 2024-03-29T11:19:46.952 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:19:46.952 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:46.952 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:46.952 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:46.952 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:46.952 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:46.952 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:19:46.952 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:19:46.952 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:19:46.952 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:46.952 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.971+0000 7f7a43fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:46.952 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.971+0000 7f7a43fff640 4 set_mon_vals no callback set 2024-03-29T11:19:46.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.972+0000 7f7a43fff640 10 monclient: shutdown 2024-03-29T11:19:46.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.972+0000 7f7a43fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:46.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.972+0000 7f7a43fff640 1 -- 172.21.15.18:0/708510266 >> v1:172.21.15.18:6789/0 conn(0x7f7a3c030660 legacy=0x7f7a3c0d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:46.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.972+0000 7f7a43fff640 1 -- 172.21.15.18:0/708510266 shutdown_connections 2024-03-29T11:19:46.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.972+0000 7f7a43fff640 1 -- 172.21.15.18:0/708510266 >> 172.21.15.18:0/708510266 conn(0x7f7a3c005c00 msgr2=0x7f7a3c006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:46.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.972+0000 7f7a43fff640 1 -- 172.21.15.18:0/708510266 shutdown_connections 2024-03-29T11:19:46.953 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.972+0000 7f7a43fff640 1 -- 172.21.15.18:0/708510266 wait complete. 2024-03-29T11:19:46.954 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.973+0000 7f7a43fff640 10 monclient: build_initial_monmap 2024-03-29T11:19:46.954 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.973+0000 7f7a43fff640 10 monclient: monmap: 2024-03-29T11:19:46.954 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:19:46.954 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:46.954 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:19:46.954 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:19:46.954 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:46.954 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:46.954 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:19:46.954 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:19:46.954 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:19:46.954 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:46.954 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.973+0000 7f7a43fff640 1 librados: starting msgr at 2024-03-29T11:19:46.954 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.973+0000 7f7a43fff640 1 librados: starting objecter 2024-03-29T11:19:46.954 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.974+0000 7f7a43fff640 1 Processor -- start 2024-03-29T11:19:46.955 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.974+0000 7f7a43fff640 1 -- start start 2024-03-29T11:19:46.955 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.974+0000 7f7a43fff640 1 librados: setting wanted keys 2024-03-29T11:19:46.955 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.974+0000 7f7a43fff640 1 librados: calling monclient init 2024-03-29T11:19:46.955 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.974+0000 7f7a43fff640 10 monclient: init 2024-03-29T11:19:46.955 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.975+0000 7f7a43fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:46.955 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.975+0000 7f7a43fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:19:46.955 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.975+0000 7f7a43fff640 10 monclient(hunting): picked mon.noname-b con 0x7f7a3c030660 addr v1:172.21.15.70:6789/0 2024-03-29T11:19:46.955 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.975+0000 7f7a43fff640 10 monclient(hunting): picked mon.noname-c con 0x7f7a3c0389e0 addr v1:172.21.15.18:6790/0 2024-03-29T11:19:46.958 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.981+0000 7f7a43fff640 10 monclient(hunting): picked mon.noname-a con 0x7f7a3c00c310 addr v1:172.21.15.18:6789/0 2024-03-29T11:19:46.958 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.981+0000 7f7a43fff640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a3c161530 con 0x7f7a3c00c310 2024-03-29T11:19:46.959 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.981+0000 7f7a43fff640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a3c00bef0 con 0x7f7a3c030660 2024-03-29T11:19:46.959 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.981+0000 7f7a43fff640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a3c009890 con 0x7f7a3c0389e0 2024-03-29T11:19:46.959 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.981+0000 7f7a43fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:46.959 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.981+0000 7f7a43fff640 10 monclient(hunting): authenticate will time out at 1413.254264s 2024-03-29T11:19:46.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.983+0000 7f7a427fc640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f7a3c0389e0 0x7f7a3c1619c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:32944/0 (socket says 172.21.15.18:32944) 2024-03-29T11:19:46.961 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.983+0000 7f7a427fc640 1 -- 172.21.15.18:0/613722404 learned_addr learned my addr 172.21.15.18:0/613722404 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:19:46.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.983+0000 7f7a2b7fe640 10 client.?.objecter ms_handle_connect 0x7f7a3c0389e0 2024-03-29T11:19:46.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.983+0000 7f7a2b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:46.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.983+0000 7f7a2b7fe640 1 -- 172.21.15.18:0/613722404 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3942027781 0 0) 0x7f7a3c009890 con 0x7f7a3c0389e0 2024-03-29T11:19:46.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.983+0000 7f7a2b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:46.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.983+0000 7f7a2b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:46.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.983+0000 7f7a2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:46.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.983+0000 7f7a2b7fe640 10 monclient(hunting): my global_id is 4685 2024-03-29T11:19:46.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.983+0000 7f7a2b7fe640 1 -- 172.21.15.18:0/613722404 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a30004eb0 con 0x7f7a3c0389e0 2024-03-29T11:19:46.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.984+0000 7f7a2b7fe640 1 -- 172.21.15.18:0/613722404 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2221443017 0 0) 0x7f7a30004eb0 con 0x7f7a3c0389e0 2024-03-29T11:19:46.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.984+0000 7f7a2b7fe640 1 -- 172.21.15.18:0/613722404 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a3c009890 con 0x7f7a3c0389e0 2024-03-29T11:19:46.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.984+0000 7f7a2b7fe640 1 -- 172.21.15.18:0/613722404 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f7a34001810 con 0x7f7a3c0389e0 2024-03-29T11:19:46.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.984+0000 7f7a2b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:46.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.984+0000 7f7a2b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:19:46.962 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.984+0000 7f7a2b7fe640 10 monclient(hunting): dump: 2024-03-29T11:19:46.963 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:19:46.963 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:46.963 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:46.963 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:46.963 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:46.963 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:46.963 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:19:46.963 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:19:46.963 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:19:46.963 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:46.963 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.984+0000 7f7a2b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:46.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.985+0000 7f7a2b7fe640 1 -- 172.21.15.18:0/613722404 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3743451647 0 0) 0x7f7a3c009890 con 0x7f7a3c0389e0 2024-03-29T11:19:46.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.985+0000 7f7a2b7fe640 1 -- 172.21.15.18:0/613722404 >> v1:172.21.15.70:6789/0 conn(0x7f7a3c030660 legacy=0x7f7a3c00a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:46.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.985+0000 7f7a2b7fe640 1 -- 172.21.15.18:0/613722404 >> v1:172.21.15.18:6789/0 conn(0x7f7a3c00c310 legacy=0x7f7a3c16be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:46.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.985+0000 7f7a2b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:46.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.985+0000 7f7a2b7fe640 1 monclient: found mon.c 2024-03-29T11:19:46.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.985+0000 7f7a2b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:46.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.985+0000 7f7a2b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:19:46.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.985+0000 7f7a2b7fe640 1 -- 172.21.15.18:0/613722404 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a3c00a070 con 0x7f7a3c0389e0 2024-03-29T11:19:46.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.985+0000 7f7a2b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:46.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.985+0000 7f7a2b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:46.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.985+0000 7f7a2b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:46.964 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.985+0000 7f7a43fff640 5 monclient: authenticate success, global_id 4685 2024-03-29T11:19:46.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.985+0000 7f7a43fff640 10 monclient: _renew_subs 2024-03-29T11:19:46.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.985+0000 7f7a43fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:19:46.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.985+0000 7f7a43fff640 1 -- 172.21.15.18:0/613722404 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a3c1620d0 con 0x7f7a3c0389e0 2024-03-29T11:19:46.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.985+0000 7f7a43fff640 10 client.4685.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:46.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.985+0000 7f7a43fff640 10 monclient: _renew_subs 2024-03-29T11:19:46.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.985+0000 7f7a43fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:19:46.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.985+0000 7f7a43fff640 1 -- 172.21.15.18:0/613722404 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a3c16d740 con 0x7f7a3c0389e0 2024-03-29T11:19:46.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.986+0000 7f7a43fff640 1 librados: init done 2024-03-29T11:19:46.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.986+0000 7f7a2b7fe640 1 -- 172.21.15.18:0/613722404 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a34001970 con 0x7f7a3c0389e0 2024-03-29T11:19:46.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.986+0000 7f7a2b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:46.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.986+0000 7f7a2b7fe640 1 -- 172.21.15.18:0/613722404 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f7a34004a40 con 0x7f7a3c0389e0 2024-03-29T11:19:46.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.986+0000 7f7a2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:46.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.986+0000 7f7a2b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:19:46.965 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.986+0000 7f7a2b7fe640 10 monclient: dump: 2024-03-29T11:19:46.965 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:19:46.966 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:46.966 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:46.966 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:46.966 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:46.966 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:46.966 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:19:46.966 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:19:46.966 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:19:46.966 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:46.966 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.986+0000 7f7a417fa640 4 set_mon_vals no callback set 2024-03-29T11:19:46.967 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.988+0000 7f7a2b7fe640 1 -- 172.21.15.18:0/613722404 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f7a34004ba0 con 0x7f7a3c0389e0 2024-03-29T11:19:46.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.990+0000 7f7a2b7fe640 1 -- 172.21.15.18:0/613722404 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3599602952 0 0) 0x7f7a34084ae0 con 0x7f7a3c0389e0 2024-03-29T11:19:46.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.990+0000 7f7a2b7fe640 10 client.4685.objecter ms_dispatch 0x7f7a44000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:19:46.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.990+0000 7f7a2b7fe640 3 client.4685.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:19:46.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.990+0000 7f7a2b7fe640 3 client.4685.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:19:46.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.990+0000 7f7a2b7fe640 20 client.4685.objecter dump_active .. 0 homeless 2024-03-29T11:19:46.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.990+0000 7f7a43fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:46.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.990+0000 7f7a43fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:46.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.990+0000 7f7a43fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:19:46.968 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.990+0000 7f7a43fff640 1 -- 172.21.15.18:0/613722404 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a24005080 con 0x7f7a3c0389e0 2024-03-29T11:19:46.970 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.992+0000 7f7a42ffd640 10 monclient: get_auth_request con 0x7f7a3006dcd0 auth_method 0 2024-03-29T11:19:46.970 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:46.993+0000 7f7a2b7fe640 10 client.4685.objecter ms_handle_connect 0x7f7a3006dcd0 2024-03-29T11:19:46.978 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:47.000+0000 7f7a2b7fe640 1 -- 172.21.15.18:0/613722404 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7a340501f0 con 0x7f7a3c0389e0 2024-03-29T11:19:46.978 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:47.000+0000 7f7a2b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:46.978 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:47.000+0000 7f7a2b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:47.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:47.366+0000 7f7a43fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:47.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:47.366+0000 7f7a43fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:47.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:47.366+0000 7f7a43fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:19:47.344 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:47.366+0000 7f7a43fff640 1 -- 172.21.15.18:0/613722404 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7a24005330 con 0x7f7a3c0389e0 2024-03-29T11:19:47.347 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:47.370+0000 7f7a2b7fe640 1 -- 172.21.15.18:0/613722404 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2084344819) 0x7f7a340522f0 con 0x7f7a3c0389e0 2024-03-29T11:19:47.347 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:47.370+0000 7f7a2b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:47.347 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:47.370+0000 7f7a2b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:47.349 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:19:47.349 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":11,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:19:08.996105+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:08.996105+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:47.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:47.374+0000 7f7a43fff640 10 librados: watch_flush enter 2024-03-29T11:19:47.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:47.375+0000 7f7a43fff640 10 librados: watch_flush exit 2024-03-29T11:19:47.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:47.375+0000 7f7a43fff640 20 client.4685.objecter shutdown clearing up homeless session... 2024-03-29T11:19:47.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:47.375+0000 7f7a43fff640 10 client.4685.objecter successfully canceled tick 2024-03-29T11:19:47.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:47.375+0000 7f7a43fff640 1 -- 172.21.15.18:0/613722404 >> v1:172.21.15.70:6816/3101871562 conn(0x7f7a3006dcd0 legacy=0x7f7a30070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:47.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:47.375+0000 7f7a43fff640 10 monclient: shutdown 2024-03-29T11:19:47.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:47.375+0000 7f7a43fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:47.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:47.375+0000 7f7a43fff640 1 -- 172.21.15.18:0/613722404 >> v1:172.21.15.18:6790/0 conn(0x7f7a3c0389e0 legacy=0x7f7a3c1619c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:47.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:47.375+0000 7f7a43fff640 1 -- 172.21.15.18:0/613722404 shutdown_connections 2024-03-29T11:19:47.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:47.375+0000 7f7a43fff640 1 -- 172.21.15.18:0/613722404 >> 172.21.15.18:0/613722404 conn(0x7f7a3c005c00 msgr2=0x7f7a3c005fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:47.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:47.376+0000 7f7a43fff640 1 -- 172.21.15.18:0/613722404 shutdown_connections 2024-03-29T11:19:47.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:47.376+0000 7f7a43fff640 1 -- 172.21.15.18:0/613722404 wait complete. 2024-03-29T11:19:47.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:47.376+0000 7f7a43fff640 1 librados: shutdown 2024-03-29T11:19:47.373 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:19:54.180 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:19:54.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.354+0000 7f527090d640 10 monclient: get_monmap_and_config 2024-03-29T11:19:54.335 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.354+0000 7f527090d640 10 monclient: build_initial_monmap 2024-03-29T11:19:54.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.354+0000 7f527090d640 10 monclient: monmap: 2024-03-29T11:19:54.336 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:19:54.336 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:54.336 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:19:54.354980+0000 2024-03-29T11:19:54.336 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:19:54.354980+0000 2024-03-29T11:19:54.336 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:54.336 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:54.336 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:19:54.336 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:19:54.336 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:19:54.336 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:54.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.354+0000 7f527090d640 1 Processor -- start 2024-03-29T11:19:54.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.355+0000 7f527090d640 1 -- start start 2024-03-29T11:19:54.336 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.355+0000 7f527090d640 10 monclient: init 2024-03-29T11:19:54.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.355+0000 7f527090d640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:54.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.355+0000 7f527090d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:19:54.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.355+0000 7f527090d640 10 monclient(hunting): picked mon.noname-a con 0x7f526c0a9750 addr v1:172.21.15.18:6789/0 2024-03-29T11:19:54.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.355+0000 7f527090d640 10 monclient(hunting): picked mon.noname-b con 0x7f526c0add40 addr v1:172.21.15.18:6790/0 2024-03-29T11:19:54.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.355+0000 7f527090d640 10 monclient(hunting): picked mon.noname-c con 0x7f526c0aad20 addr v1:172.21.15.70:6789/0 2024-03-29T11:19:54.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.355+0000 7f527090d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f526c149870 con 0x7f526c0a9750 2024-03-29T11:19:54.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.356+0000 7f526ad9d640 1 --1- >> v1:172.21.15.18:6789/0 conn(0x7f526c0a9750 0x7f526c149fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.18:6789/0 says I am v1:172.21.15.18:43282/0 (socket says 172.21.15.18:43282) 2024-03-29T11:19:54.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.356+0000 7f526ad9d640 1 -- 172.21.15.18:0/2191977272 learned_addr learned my addr 172.21.15.18:0/2191977272 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:19:54.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.356+0000 7f527090d640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f526c0abf80 con 0x7f526c0aad20 2024-03-29T11:19:54.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.356+0000 7f527090d640 1 -- 172.21.15.18:0/2191977272 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f526c149ac0 con 0x7f526c0add40 2024-03-29T11:19:54.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.356+0000 7f527090d640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:54.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.356+0000 7f527090d640 10 monclient(hunting): authenticate will time out at 1420.628093s 2024-03-29T11:19:54.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.356+0000 7f5269d9b640 1 -- 172.21.15.18:0/2191977272 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2079818287 0 0) 0x7f526c149870 con 0x7f526c0a9750 2024-03-29T11:19:54.337 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.356+0000 7f5269d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:19:54.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.356+0000 7f5269d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:54.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.356+0000 7f5269d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:54.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.356+0000 7f5269d9b640 10 monclient(hunting): my global_id is 4590 2024-03-29T11:19:54.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.356+0000 7f5269d9b640 1 -- 172.21.15.18:0/2191977272 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5254003520 con 0x7f526c0a9750 2024-03-29T11:19:54.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.357+0000 7f5269d9b640 1 -- 172.21.15.18:0/2191977272 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3574157404 0 0) 0x7f5254003520 con 0x7f526c0a9750 2024-03-29T11:19:54.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.357+0000 7f5269d9b640 1 -- 172.21.15.18:0/2191977272 >> v1:172.21.15.18:6790/0 conn(0x7f526c0add40 legacy=0x7f526c143610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:54.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.357+0000 7f5269d9b640 1 -- 172.21.15.18:0/2191977272 >> v1:172.21.15.70:6789/0 conn(0x7f526c0aad20 legacy=0x7f526c151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:54.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.357+0000 7f5269d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:19:54.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.357+0000 7f5269d9b640 1 monclient: found mon.noname-a 2024-03-29T11:19:54.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.357+0000 7f5269d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:54.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.357+0000 7f5269d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-29T11:19:54.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.357+0000 7f5269d9b640 1 -- 172.21.15.18:0/2191977272 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f526c14a6f0 con 0x7f526c0a9750 2024-03-29T11:19:54.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.357+0000 7f5269d9b640 10 monclient: _finish_auth 0 2024-03-29T11:19:54.338 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.357+0000 7f5269d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:19:54.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.357+0000 7f5269d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:54.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.357+0000 7f5269d9b640 1 -- 172.21.15.18:0/2191977272 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f525c001690 con 0x7f526c0a9750 2024-03-29T11:19:54.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.357+0000 7f5269d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:54.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.357+0000 7f5269d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:54.339 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.357+0000 7f5269d9b640 10 monclient: dump: 2024-03-29T11:19:54.339 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:19:54.339 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:54.339 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:54.339 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:54.339 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:54.339 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:54.339 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:19:54.339 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:19:54.339 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:19:54.339 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:54.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.357+0000 7f527090d640 5 monclient: authenticate success, global_id 4590 2024-03-29T11:19:54.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.357+0000 7f527090d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:19:54.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.357+0000 7f5269d9b640 1 -- 172.21.15.18:0/2191977272 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f525c001860 con 0x7f526c0a9750 2024-03-29T11:19:54.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.357+0000 7f5269d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:54.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.357+0000 7f5269d9b640 1 -- 172.21.15.18:0/2191977272 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f525c001c70 con 0x7f526c0a9750 2024-03-29T11:19:54.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.357+0000 7f5269d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:54.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.357+0000 7f5269d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:54.340 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.357+0000 7f5269d9b640 10 monclient: dump: 2024-03-29T11:19:54.340 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:19:54.340 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:54.340 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:54.340 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:54.340 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:54.340 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:54.341 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:19:54.341 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:19:54.341 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:19:54.341 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:54.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.357+0000 7f527090d640 10 monclient: get_monmap_and_config success 2024-03-29T11:19:54.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.357+0000 7f527090d640 4 set_mon_vals no callback set 2024-03-29T11:19:54.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.358+0000 7f527090d640 10 monclient: shutdown 2024-03-29T11:19:54.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.358+0000 7f527090d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:54.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.358+0000 7f527090d640 1 -- 172.21.15.18:0/2191977272 >> v1:172.21.15.18:6789/0 conn(0x7f526c0a9750 legacy=0x7f526c149fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:54.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.359+0000 7f527090d640 1 -- 172.21.15.18:0/2191977272 shutdown_connections 2024-03-29T11:19:54.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.359+0000 7f527090d640 1 -- 172.21.15.18:0/2191977272 >> 172.21.15.18:0/2191977272 conn(0x7f526c0673e0 msgr2=0x7f526c0b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:54.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.359+0000 7f527090d640 1 -- 172.21.15.18:0/2191977272 shutdown_connections 2024-03-29T11:19:54.341 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.359+0000 7f527090d640 1 -- 172.21.15.18:0/2191977272 wait complete. 2024-03-29T11:19:54.349 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.360+0000 7f527090d640 10 monclient: build_initial_monmap 2024-03-29T11:19:54.349 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.361+0000 7f527090d640 10 monclient: monmap: 2024-03-29T11:19:54.349 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:19:54.349 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:19:54.349 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:19:54.349 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:19:54.349 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:19:54.349 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:54.349 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:19:54.350 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:19:54.350 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:19:54.350 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:54.350 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.361+0000 7f527090d640 1 librados: starting msgr at 2024-03-29T11:19:54.350 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.361+0000 7f527090d640 1 librados: starting objecter 2024-03-29T11:19:54.350 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.362+0000 7f527090d640 1 Processor -- start 2024-03-29T11:19:54.350 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.362+0000 7f527090d640 1 -- start start 2024-03-29T11:19:54.350 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.362+0000 7f527090d640 1 librados: setting wanted keys 2024-03-29T11:19:54.350 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.362+0000 7f527090d640 1 librados: calling monclient init 2024-03-29T11:19:54.350 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.362+0000 7f527090d640 10 monclient: init 2024-03-29T11:19:54.350 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.363+0000 7f527090d640 10 monclient: _reopen_session rank -1 2024-03-29T11:19:54.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.363+0000 7f527090d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:19:54.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.364+0000 7f527090d640 10 monclient(hunting): picked mon.noname-c con 0x7f526c0a9750 addr v1:172.21.15.18:6790/0 2024-03-29T11:19:54.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.364+0000 7f527090d640 10 monclient(hunting): picked mon.noname-b con 0x7f526c0aad20 addr v1:172.21.15.70:6789/0 2024-03-29T11:19:54.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.364+0000 7f527090d640 10 monclient(hunting): picked mon.noname-a con 0x7f526c0add40 addr v1:172.21.15.18:6789/0 2024-03-29T11:19:54.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.364+0000 7f527090d640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f526c1478c0 con 0x7f526c0add40 2024-03-29T11:19:54.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.364+0000 7f526ad9d640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f526c0a9750 0x7f526c147f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:49616/0 (socket says 172.21.15.18:49616) 2024-03-29T11:19:54.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.364+0000 7f526ad9d640 1 -- 172.21.15.18:0/687448283 learned_addr learned my addr 172.21.15.18:0/687448283 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:19:54.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.364+0000 7f527090d640 1 -- 172.21.15.18:0/687448283 --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f526c1deab0 con 0x7f526c0aad20 2024-03-29T11:19:54.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.364+0000 7f527090d640 1 -- 172.21.15.18:0/687448283 --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f526c1486a0 con 0x7f526c0a9750 2024-03-29T11:19:54.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.364+0000 7f527090d640 10 monclient(hunting): _renew_subs 2024-03-29T11:19:54.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.364+0000 7f527090d640 10 monclient(hunting): authenticate will time out at 1420.636657s 2024-03-29T11:19:54.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.364+0000 7f52537fe640 10 client.?.objecter ms_handle_connect 0x7f526c0add40 2024-03-29T11:19:54.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.364+0000 7f52537fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:54.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.364+0000 7f52537fe640 10 client.?.objecter ms_handle_connect 0x7f526c0aad20 2024-03-29T11:19:54.351 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.364+0000 7f52537fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:54.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.364+0000 7f52537fe640 10 client.?.objecter ms_handle_connect 0x7f526c0a9750 2024-03-29T11:19:54.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.365+0000 7f52537fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:19:54.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.365+0000 7f52537fe640 1 -- 172.21.15.18:0/687448283 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3184380850 0 0) 0x7f526c1478c0 con 0x7f526c0add40 2024-03-29T11:19:54.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.365+0000 7f52537fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:54.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.365+0000 7f52537fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:54.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.365+0000 7f52537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:54.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.365+0000 7f52537fe640 10 monclient(hunting): my global_id is 4593 2024-03-29T11:19:54.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.365+0000 7f52537fe640 1 -- 172.21.15.18:0/687448283 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5258004eb0 con 0x7f526c0add40 2024-03-29T11:19:54.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.365+0000 7f52537fe640 1 -- 172.21.15.18:0/687448283 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2484809332 0 0) 0x7f526c1deab0 con 0x7f526c0aad20 2024-03-29T11:19:54.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.365+0000 7f52537fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:54.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.365+0000 7f52537fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:54.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.365+0000 7f52537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:54.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.365+0000 7f52537fe640 10 monclient(hunting): my global_id is 4540 2024-03-29T11:19:54.352 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.365+0000 7f52537fe640 1 -- 172.21.15.18:0/687448283 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f526c1478c0 con 0x7f526c0aad20 2024-03-29T11:19:54.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.365+0000 7f52537fe640 1 -- 172.21.15.18:0/687448283 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3756291884 0 0) 0x7f526c1486a0 con 0x7f526c0a9750 2024-03-29T11:19:54.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.365+0000 7f52537fe640 10 monclient(hunting): _negotiate 2024-03-29T11:19:54.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.365+0000 7f52537fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:19:54.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.365+0000 7f52537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:19:54.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.365+0000 7f52537fe640 10 monclient(hunting): my global_id is 4688 2024-03-29T11:19:54.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.365+0000 7f52537fe640 1 -- 172.21.15.18:0/687448283 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f526c1deab0 con 0x7f526c0a9750 2024-03-29T11:19:54.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.365+0000 7f52537fe640 1 -- 172.21.15.18:0/687448283 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3321759384 0 0) 0x7f5258004eb0 con 0x7f526c0add40 2024-03-29T11:19:54.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.365+0000 7f52537fe640 1 -- 172.21.15.18:0/687448283 --> v1:172.21.15.18:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f526c1486a0 con 0x7f526c0add40 2024-03-29T11:19:54.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.365+0000 7f52537fe640 1 -- 172.21.15.18:0/687448283 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f5264001c30 con 0x7f526c0add40 2024-03-29T11:19:54.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.365+0000 7f52537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:54.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.365+0000 7f52537fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:19:54.353 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.365+0000 7f52537fe640 10 monclient(hunting): dump: 2024-03-29T11:19:54.353 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:19:54.353 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:54.354 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:54.354 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:54.354 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:54.354 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:54.354 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:19:54.354 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:19:54.354 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:19:54.354 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:54.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.365+0000 7f52537fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:19:54.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.365+0000 7f52537fe640 1 -- 172.21.15.18:0/687448283 <== mon.1 v1:172.21.15.70:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1149022491 0 0) 0x7f526c1478c0 con 0x7f526c0aad20 2024-03-29T11:19:54.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.366+0000 7f52537fe640 1 -- 172.21.15.18:0/687448283 --> v1:172.21.15.70:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5258004eb0 con 0x7f526c0aad20 2024-03-29T11:19:54.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.366+0000 7f52537fe640 1 -- 172.21.15.18:0/687448283 <== mon.1 v1:172.21.15.70:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f52600016b0 con 0x7f526c0aad20 2024-03-29T11:19:54.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.366+0000 7f52537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:54.354 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.366+0000 7f52537fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:19:54.355 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.366+0000 7f52537fe640 10 monclient(hunting): dump: 2024-03-29T11:19:54.355 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:19:54.355 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:54.355 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:54.355 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:54.355 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:54.355 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:54.355 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:19:54.355 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:19:54.355 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:19:54.355 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:54.355 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.366+0000 7f52537fe640 1 -- 172.21.15.18:0/687448283 <== mon.0 v1:172.21.15.18:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3904650169 0 0) 0x7f526c1486a0 con 0x7f526c0add40 2024-03-29T11:19:54.355 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.366+0000 7f52537fe640 1 -- 172.21.15.18:0/687448283 >> v1:172.21.15.18:6790/0 conn(0x7f526c0a9750 legacy=0x7f526c147f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:54.355 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.366+0000 7f52537fe640 1 -- 172.21.15.18:0/687448283 >> v1:172.21.15.70:6789/0 conn(0x7f526c0aad20 legacy=0x7f526c1defd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:54.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.366+0000 7f52537fe640 10 monclient: _finish_hunting 0 2024-03-29T11:19:54.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.366+0000 7f52537fe640 1 monclient: found mon.a 2024-03-29T11:19:54.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.366+0000 7f52537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:19:54.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.366+0000 7f52537fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:19:54.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.366+0000 7f52537fe640 1 -- 172.21.15.18:0/687448283 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f526c1df6e0 con 0x7f526c0add40 2024-03-29T11:19:54.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.366+0000 7f52537fe640 10 monclient: _finish_auth 0 2024-03-29T11:19:54.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.366+0000 7f52537fe640 10 monclient: _check_auth_tickets 2024-03-29T11:19:54.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.366+0000 7f52537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:19:54.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.366+0000 7f527090d640 5 monclient: authenticate success, global_id 4593 2024-03-29T11:19:54.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.366+0000 7f527090d640 10 monclient: _renew_subs 2024-03-29T11:19:54.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.366+0000 7f527090d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:19:54.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.366+0000 7f527090d640 1 -- 172.21.15.18:0/687448283 --> v1:172.21.15.18:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f526c144740 con 0x7f526c0add40 2024-03-29T11:19:54.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.366+0000 7f527090d640 10 client.4593.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:19:54.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.366+0000 7f527090d640 10 monclient: _renew_subs 2024-03-29T11:19:54.356 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.366+0000 7f527090d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:19:54.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.366+0000 7f527090d640 1 -- 172.21.15.18:0/687448283 --> v1:172.21.15.18:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f526c1474d0 con 0x7f526c0add40 2024-03-29T11:19:54.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.366+0000 7f527090d640 1 librados: init done 2024-03-29T11:19:54.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.367+0000 7f52537fe640 1 -- 172.21.15.18:0/687448283 <== mon.0 v1:172.21.15.18:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5264003970 con 0x7f526c0add40 2024-03-29T11:19:54.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.367+0000 7f52537fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:19:54.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.367+0000 7f52537fe640 1 -- 172.21.15.18:0/687448283 <== mon.0 v1:172.21.15.18:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f5264003d00 con 0x7f526c0add40 2024-03-29T11:19:54.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.367+0000 7f52537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:19:54.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.367+0000 7f526959a640 4 set_mon_vals no callback set 2024-03-29T11:19:54.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.367+0000 7f52537fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:19:54.357 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.367+0000 7f52537fe640 10 monclient: dump: 2024-03-29T11:19:54.357 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:19:54.357 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:19:54.357 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:54.357 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:19:54.357 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:19:54.357 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:19:54.358 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:19:54.358 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:19:54.358 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:19:54.358 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:19:54.358 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.368+0000 7f527090d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:54.358 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.368+0000 7f527090d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:54.358 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.368+0000 7f527090d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:19:54.358 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.368+0000 7f527090d640 1 -- 172.21.15.18:0/687448283 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f524c005080 con 0x7f526c0add40 2024-03-29T11:19:54.358 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.369+0000 7f52537fe640 1 -- 172.21.15.18:0/687448283 <== mon.0 v1:172.21.15.18:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f5264003e60 con 0x7f526c0add40 2024-03-29T11:19:54.358 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.370+0000 7f52537fe640 1 -- 172.21.15.18:0/687448283 <== mon.0 v1:172.21.15.18:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3599602952 0 0) 0x7f5264083be0 con 0x7f526c0add40 2024-03-29T11:19:54.358 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.370+0000 7f52537fe640 10 client.4593.objecter ms_dispatch 0x7f526c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:19:54.358 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.370+0000 7f52537fe640 3 client.4593.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:19:54.358 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.370+0000 7f52537fe640 3 client.4593.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:19:54.358 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.370+0000 7f52537fe640 20 client.4593.objecter dump_active .. 0 homeless 2024-03-29T11:19:54.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.370+0000 7f526ad9d640 10 monclient: get_auth_request con 0x7f525806dfe0 auth_method 0 2024-03-29T11:19:54.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.371+0000 7f52537fe640 10 client.4593.objecter ms_handle_connect 0x7f525806dfe0 2024-03-29T11:19:54.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.378+0000 7f52537fe640 1 -- 172.21.15.18:0/687448283 <== mon.0 v1:172.21.15.18:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f526404f4c0 con 0x7f526c0add40 2024-03-29T11:19:54.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.378+0000 7f52537fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:19:54.359 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.378+0000 7f52537fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:19:54.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.700+0000 7f527090d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:54.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.700+0000 7f527090d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:54.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.700+0000 7f527090d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.18:6789/0 2024-03-29T11:19:54.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.700+0000 7f527090d640 1 -- 172.21.15.18:0/687448283 --> v1:172.21.15.18:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f524c005330 con 0x7f526c0add40 2024-03-29T11:19:54.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.701+0000 7f52537fe640 1 -- 172.21.15.18:0/687448283 <== mon.0 v1:172.21.15.18:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2084344819) 0x7f52640513f0 con 0x7f526c0add40 2024-03-29T11:19:54.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.701+0000 7f52537fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:19:54.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.701+0000 7f52537fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:19:54.678 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:19:54.678 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":11,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:19:08.996105+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:08.996105+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:19:54.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.704+0000 7f527090d640 10 librados: watch_flush enter 2024-03-29T11:19:54.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.704+0000 7f527090d640 10 librados: watch_flush exit 2024-03-29T11:19:54.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.704+0000 7f527090d640 20 client.4593.objecter shutdown clearing up homeless session... 2024-03-29T11:19:54.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.704+0000 7f527090d640 10 client.4593.objecter successfully canceled tick 2024-03-29T11:19:54.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.704+0000 7f527090d640 1 -- 172.21.15.18:0/687448283 >> v1:172.21.15.70:6816/3101871562 conn(0x7f525806dfe0 legacy=0x7f52580704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:54.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.704+0000 7f527090d640 10 monclient: shutdown 2024-03-29T11:19:54.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.704+0000 7f527090d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:19:54.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.704+0000 7f527090d640 1 -- 172.21.15.18:0/687448283 >> v1:172.21.15.18:6789/0 conn(0x7f526c0add40 legacy=0x7f526c13c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:19:54.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.704+0000 7f527090d640 1 -- 172.21.15.18:0/687448283 shutdown_connections 2024-03-29T11:19:54.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.704+0000 7f527090d640 1 -- 172.21.15.18:0/687448283 >> 172.21.15.18:0/687448283 conn(0x7f526c0673e0 msgr2=0x7f526c141940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:19:54.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.704+0000 7f527090d640 1 -- 172.21.15.18:0/687448283 shutdown_connections 2024-03-29T11:19:54.681 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.704+0000 7f527090d640 1 -- 172.21.15.18:0/687448283 wait complete. 2024-03-29T11:19:54.682 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:19:54.704+0000 7f527090d640 1 librados: shutdown 2024-03-29T11:19:54.697 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:01.504 DEBUG:teuthology.orchestra.run.smithi018:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:01.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.682+0000 7f43714b3640 10 monclient: get_monmap_and_config 2024-03-29T11:20:01.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.682+0000 7f43714b3640 10 monclient: build_initial_monmap 2024-03-29T11:20:01.659 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.682+0000 7f43714b3640 10 monclient: monmap: 2024-03-29T11:20:01.659 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:20:01.660 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:20:01.660 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:20:01.683687+0000 2024-03-29T11:20:01.660 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:20:01.683687+0000 2024-03-29T11:20:01.660 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:01.660 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:20:01.660 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:20:01.660 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-c 2024-03-29T11:20:01.660 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-b 2024-03-29T11:20:01.660 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:20:01.660 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.683+0000 7f43714b3640 1 Processor -- start 2024-03-29T11:20:01.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.683+0000 7f43714b3640 1 -- start start 2024-03-29T11:20:01.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.683+0000 7f43714b3640 10 monclient: init 2024-03-29T11:20:01.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.684+0000 7f43714b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:01.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.684+0000 7f43714b3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:20:01.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.684+0000 7f43714b3640 10 monclient(hunting): picked mon.noname-b con 0x7f436c0b1f60 addr v1:172.21.15.18:6790/0 2024-03-29T11:20:01.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.684+0000 7f43714b3640 10 monclient(hunting): picked mon.noname-c con 0x7f436c071eb0 addr v1:172.21.15.70:6789/0 2024-03-29T11:20:01.661 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.684+0000 7f43714b3640 10 monclient(hunting): picked mon.noname-a con 0x7f436c143d10 addr v1:172.21.15.18:6789/0 2024-03-29T11:20:01.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.684+0000 7f43714b3640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f436c06ef80 con 0x7f436c143d10 2024-03-29T11:20:01.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.684+0000 7f43714b3640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f436c06f260 con 0x7f436c071eb0 2024-03-29T11:20:01.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.684+0000 7f43714b3640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f436c0b40c0 con 0x7f436c0b1f60 2024-03-29T11:20:01.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.684+0000 7f43714b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:01.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.685+0000 7f43714b3640 10 monclient(hunting): authenticate will time out at 1427.956919s 2024-03-29T11:20:01.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.685+0000 7f43627fc640 1 --1- >> v1:172.21.15.70:6789/0 conn(0x7f436c071eb0 0x7f436c146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.70:6789/0 says I am v1:172.21.15.18:37450/0 (socket says 172.21.15.18:37450) 2024-03-29T11:20:01.662 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.685+0000 7f43627fc640 1 -- 172.21.15.18:0/1659414451 learned_addr learned my addr 172.21.15.18:0/1659414451 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:20:01.663 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.685+0000 7f436a7fc640 1 -- 172.21.15.18:0/1659414451 <== mon.1 v1:172.21.15.70:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 402075815 0 0) 0x7f436c06f260 con 0x7f436c071eb0 2024-03-29T11:20:01.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.685+0000 7f436a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:01.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.685+0000 7f436a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:01.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.685+0000 7f436a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:01.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.685+0000 7f436a7fc640 10 monclient(hunting): my global_id is 4543 2024-03-29T11:20:01.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.685+0000 7f436a7fc640 1 -- 172.21.15.18:0/1659414451 --> v1:172.21.15.70:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4358003520 con 0x7f436c071eb0 2024-03-29T11:20:01.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.685+0000 7f436a7fc640 1 -- 172.21.15.18:0/1659414451 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2334504722 0 0) 0x7f436c06ef80 con 0x7f436c143d10 2024-03-29T11:20:01.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.685+0000 7f436a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:01.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.685+0000 7f436a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:01.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.685+0000 7f436a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:01.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.685+0000 7f436a7fc640 10 monclient(hunting): my global_id is 4596 2024-03-29T11:20:01.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.685+0000 7f436a7fc640 1 -- 172.21.15.18:0/1659414451 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f436c06f260 con 0x7f436c143d10 2024-03-29T11:20:01.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.685+0000 7f436a7fc640 1 -- 172.21.15.18:0/1659414451 <== mon.0 v1:172.21.15.18:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1994482524 0 0) 0x7f436c06f260 con 0x7f436c143d10 2024-03-29T11:20:01.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.685+0000 7f436a7fc640 1 -- 172.21.15.18:0/1659414451 >> v1:172.21.15.18:6790/0 conn(0x7f436c0b1f60 legacy=0x7f436c143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:20:01.664 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.685+0000 7f436a7fc640 1 -- 172.21.15.18:0/1659414451 >> v1:172.21.15.70:6789/0 conn(0x7f436c071eb0 legacy=0x7f436c146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:01.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.685+0000 7f436a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:20:01.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.685+0000 7f436a7fc640 1 monclient: found mon.noname-a 2024-03-29T11:20:01.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.685+0000 7f436a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:01.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.685+0000 7f436a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.18:6789/0 2024-03-29T11:20:01.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.685+0000 7f436a7fc640 1 -- 172.21.15.18:0/1659414451 --> v1:172.21.15.18:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f436c155d70 con 0x7f436c143d10 2024-03-29T11:20:01.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.685+0000 7f436a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:20:01.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.685+0000 7f436a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:20:01.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.685+0000 7f436a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:01.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.686+0000 7f43714b3640 5 monclient: authenticate success, global_id 4596 2024-03-29T11:20:01.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.686+0000 7f43714b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:01.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.686+0000 7f436a7fc640 1 -- 172.21.15.18:0/1659414451 <== mon.0 v1:172.21.15.18:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f4364001630 con 0x7f436c143d10 2024-03-29T11:20:01.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.686+0000 7f436a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:01.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.686+0000 7f436a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:01.665 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.686+0000 7f436a7fc640 10 monclient: dump: 2024-03-29T11:20:01.665 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:20:01.666 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:20:01.666 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:20:01.666 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:20:01.666 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:01.666 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:20:01.666 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:20:01.666 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:20:01.666 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:20:01.666 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:20:01.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.686+0000 7f43714b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:01.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.686+0000 7f436a7fc640 1 -- 172.21.15.18:0/1659414451 <== mon.0 v1:172.21.15.18:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43640018c0 con 0x7f436c143d10 2024-03-29T11:20:01.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.686+0000 7f436a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:01.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.686+0000 7f43714b3640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:01.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.686+0000 7f43714b3640 4 set_mon_vals no callback set 2024-03-29T11:20:01.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.686+0000 7f436a7fc640 1 -- 172.21.15.18:0/1659414451 <== mon.0 v1:172.21.15.18:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f4364001c50 con 0x7f436c143d10 2024-03-29T11:20:01.666 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.686+0000 7f436a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:01.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.686+0000 7f436a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:01.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.686+0000 7f436a7fc640 10 monclient: dump: 2024-03-29T11:20:01.667 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:20:01.667 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:20:01.667 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:20:01.667 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:20:01.667 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:01.667 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:20:01.667 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:20:01.667 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:20:01.667 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:20:01.667 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:20:01.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.687+0000 7f43714b3640 10 monclient: shutdown 2024-03-29T11:20:01.667 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.687+0000 7f43714b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:01.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.687+0000 7f43714b3640 1 -- 172.21.15.18:0/1659414451 >> v1:172.21.15.18:6789/0 conn(0x7f436c143d10 legacy=0x7f436c146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:01.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.687+0000 7f43714b3640 1 -- 172.21.15.18:0/1659414451 shutdown_connections 2024-03-29T11:20:01.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.687+0000 7f43714b3640 1 -- 172.21.15.18:0/1659414451 >> 172.21.15.18:0/1659414451 conn(0x7f436c066a00 msgr2=0x7f436c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:01.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.687+0000 7f43714b3640 1 -- 172.21.15.18:0/1659414451 shutdown_connections 2024-03-29T11:20:01.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.687+0000 7f43714b3640 1 -- 172.21.15.18:0/1659414451 wait complete. 2024-03-29T11:20:01.670 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.688+0000 7f43714b3640 10 monclient: build_initial_monmap 2024-03-29T11:20:01.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.688+0000 7f43714b3640 10 monclient: monmap: 2024-03-29T11:20:01.671 INFO:teuthology.misc.health.smithi018.stderr:epoch 0 2024-03-29T11:20:01.671 INFO:teuthology.misc.health.smithi018.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:01.671 INFO:teuthology.misc.health.smithi018.stderr:last_changed 0.000000 2024-03-29T11:20:01.671 INFO:teuthology.misc.health.smithi018.stderr:created 0.000000 2024-03-29T11:20:01.671 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:01.671 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:20:01.671 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.noname-a 2024-03-29T11:20:01.671 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.noname-b 2024-03-29T11:20:01.671 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.noname-c 2024-03-29T11:20:01.671 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:20:01.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.689+0000 7f43714b3640 1 librados: starting msgr at 2024-03-29T11:20:01.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.689+0000 7f43714b3640 1 librados: starting objecter 2024-03-29T11:20:01.671 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.689+0000 7f43714b3640 1 Processor -- start 2024-03-29T11:20:01.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.689+0000 7f43714b3640 1 -- start start 2024-03-29T11:20:01.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.689+0000 7f43714b3640 1 librados: setting wanted keys 2024-03-29T11:20:01.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.689+0000 7f43714b3640 1 librados: calling monclient init 2024-03-29T11:20:01.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.689+0000 7f43714b3640 10 monclient: init 2024-03-29T11:20:01.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.690+0000 7f43714b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:01.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.690+0000 7f43714b3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:20:01.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.690+0000 7f43714b3640 10 monclient(hunting): picked mon.noname-c con 0x7f436c0b1f60 addr v1:172.21.15.18:6790/0 2024-03-29T11:20:01.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.690+0000 7f43714b3640 10 monclient(hunting): picked mon.noname-b con 0x7f436c143d10 addr v1:172.21.15.70:6789/0 2024-03-29T11:20:01.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.690+0000 7f43714b3640 10 monclient(hunting): picked mon.noname-a con 0x7f436c1493d0 addr v1:172.21.15.18:6789/0 2024-03-29T11:20:01.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.690+0000 7f43714b3640 1 -- --> v1:172.21.15.18:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f436c14ad60 con 0x7f436c1493d0 2024-03-29T11:20:01.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.690+0000 7f43714b3640 1 -- --> v1:172.21.15.70:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f436c13ba60 con 0x7f436c143d10 2024-03-29T11:20:01.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.690+0000 7f43714b3640 1 -- --> v1:172.21.15.18:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f436c14afb0 con 0x7f436c0b1f60 2024-03-29T11:20:01.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.690+0000 7f43714b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:01.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.690+0000 7f43714b3640 10 monclient(hunting): authenticate will time out at 1427.962382s 2024-03-29T11:20:01.672 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.690+0000 7f436affd640 1 --1- >> v1:172.21.15.18:6790/0 conn(0x7f436c0b1f60 0x7f436c13f6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.18:6790/0 says I am v1:172.21.15.18:38194/0 (socket says 172.21.15.18:38194) 2024-03-29T11:20:01.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.690+0000 7f436affd640 1 -- 172.21.15.18:0/2892437054 learned_addr learned my addr 172.21.15.18:0/2892437054 (peer_addr_for_me v1:172.21.15.18:0/0) 2024-03-29T11:20:01.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.690+0000 7f4363fff640 10 client.?.objecter ms_handle_connect 0x7f436c0b1f60 2024-03-29T11:20:01.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.690+0000 7f4363fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:01.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.691+0000 7f4363fff640 10 client.?.objecter ms_handle_connect 0x7f436c1493d0 2024-03-29T11:20:01.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.691+0000 7f4363fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:01.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.691+0000 7f4363fff640 1 -- 172.21.15.18:0/2892437054 <== mon.2 v1:172.21.15.18:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3851840719 0 0) 0x7f436c14afb0 con 0x7f436c0b1f60 2024-03-29T11:20:01.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.691+0000 7f4363fff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:01.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.691+0000 7f4363fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:01.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.691+0000 7f4363fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:01.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.691+0000 7f4363fff640 10 monclient(hunting): my global_id is 4691 2024-03-29T11:20:01.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.691+0000 7f4363fff640 1 -- 172.21.15.18:0/2892437054 --> v1:172.21.15.18:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f434c004eb0 con 0x7f436c0b1f60 2024-03-29T11:20:01.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.691+0000 7f4363fff640 1 -- 172.21.15.18:0/2892437054 <== mon.0 v1:172.21.15.18:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 719831403 0 0) 0x7f436c14ad60 con 0x7f436c1493d0 2024-03-29T11:20:01.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.691+0000 7f4363fff640 10 monclient(hunting): _negotiate 2024-03-29T11:20:01.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.691+0000 7f4363fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:01.673 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.691+0000 7f4363fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:01.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.691+0000 7f4363fff640 10 monclient(hunting): my global_id is 4599 2024-03-29T11:20:01.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.691+0000 7f4363fff640 1 -- 172.21.15.18:0/2892437054 --> v1:172.21.15.18:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f436c14afb0 con 0x7f436c1493d0 2024-03-29T11:20:01.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.691+0000 7f4363fff640 1 -- 172.21.15.18:0/2892437054 <== mon.2 v1:172.21.15.18:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4138531038 0 0) 0x7f434c004eb0 con 0x7f436c0b1f60 2024-03-29T11:20:01.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.691+0000 7f4363fff640 1 -- 172.21.15.18:0/2892437054 --> v1:172.21.15.18:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f436c14ad60 con 0x7f436c0b1f60 2024-03-29T11:20:01.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.691+0000 7f4363fff640 1 -- 172.21.15.18:0/2892437054 <== mon.2 v1:172.21.15.18:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f435c003f60 con 0x7f436c0b1f60 2024-03-29T11:20:01.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.691+0000 7f4363fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:01.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.691+0000 7f4363fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:01.674 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.691+0000 7f4363fff640 10 monclient(hunting): dump: 2024-03-29T11:20:01.674 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:20:01.674 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:20:01.674 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:20:01.674 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:20:01.674 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:01.674 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:20:01.674 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:20:01.675 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:20:01.675 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:20:01.675 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:20:01.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.691+0000 7f4363fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:01.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.692+0000 7f4363fff640 1 -- 172.21.15.18:0/2892437054 <== mon.2 v1:172.21.15.18:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1016381398 0 0) 0x7f436c14ad60 con 0x7f436c0b1f60 2024-03-29T11:20:01.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.692+0000 7f4363fff640 1 -- 172.21.15.18:0/2892437054 >> v1:172.21.15.70:6789/0 conn(0x7f436c143d10 legacy=0x7f436c13ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:01.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.692+0000 7f4363fff640 1 -- 172.21.15.18:0/2892437054 >> v1:172.21.15.18:6789/0 conn(0x7f436c1493d0 legacy=0x7f436c13b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:01.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.692+0000 7f4363fff640 10 monclient: _finish_hunting 0 2024-03-29T11:20:01.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.692+0000 7f4363fff640 1 monclient: found mon.c 2024-03-29T11:20:01.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.692+0000 7f4363fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:01.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.692+0000 7f4363fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:20:01.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.692+0000 7f4363fff640 1 -- 172.21.15.18:0/2892437054 --> v1:172.21.15.18:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f436c14b290 con 0x7f436c0b1f60 2024-03-29T11:20:01.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.692+0000 7f4363fff640 10 monclient: _finish_auth 0 2024-03-29T11:20:01.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.692+0000 7f4363fff640 10 monclient: _check_auth_tickets 2024-03-29T11:20:01.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.692+0000 7f4363fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:01.675 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.692+0000 7f43714b3640 5 monclient: authenticate success, global_id 4691 2024-03-29T11:20:01.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.692+0000 7f43714b3640 10 monclient: _renew_subs 2024-03-29T11:20:01.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.692+0000 7f43714b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:20:01.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.692+0000 7f43714b3640 1 -- 172.21.15.18:0/2892437054 --> v1:172.21.15.18:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f436c149ae0 con 0x7f436c0b1f60 2024-03-29T11:20:01.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.692+0000 7f43714b3640 10 client.4691.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:01.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.692+0000 7f43714b3640 10 monclient: _renew_subs 2024-03-29T11:20:01.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.692+0000 7f43714b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:20:01.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.692+0000 7f43714b3640 1 -- 172.21.15.18:0/2892437054 --> v1:172.21.15.18:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f436c154df0 con 0x7f436c0b1f60 2024-03-29T11:20:01.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.692+0000 7f43714b3640 1 librados: init done 2024-03-29T11:20:01.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.693+0000 7f4363fff640 1 -- 172.21.15.18:0/2892437054 <== mon.2 v1:172.21.15.18:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f435c004470 con 0x7f436c0b1f60 2024-03-29T11:20:01.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.693+0000 7f4363fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:01.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.693+0000 7f4363fff640 1 -- 172.21.15.18:0/2892437054 <== mon.2 v1:172.21.15.18:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 3002408706 0 0) 0x7f435c001070 con 0x7f436c0b1f60 2024-03-29T11:20:01.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.693+0000 7f4363fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:01.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.693+0000 7f4363fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:01.676 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.693+0000 7f4363fff640 10 monclient: dump: 2024-03-29T11:20:01.676 INFO:teuthology.misc.health.smithi018.stderr:epoch 1 2024-03-29T11:20:01.676 INFO:teuthology.misc.health.smithi018.stderr:fsid 303809eb-4027-4a6a-aa53-d48fafbbf1ea 2024-03-29T11:20:01.677 INFO:teuthology.misc.health.smithi018.stderr:last_changed 2024-03-29T11:08:29.824471+0000 2024-03-29T11:20:01.677 INFO:teuthology.misc.health.smithi018.stderr:created 2024-03-29T11:08:29.824471+0000 2024-03-29T11:20:01.677 INFO:teuthology.misc.health.smithi018.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:01.677 INFO:teuthology.misc.health.smithi018.stderr:election_strategy: 1 2024-03-29T11:20:01.677 INFO:teuthology.misc.health.smithi018.stderr:0: v1:172.21.15.18:6789/0 mon.a 2024-03-29T11:20:01.677 INFO:teuthology.misc.health.smithi018.stderr:1: v1:172.21.15.70:6789/0 mon.b 2024-03-29T11:20:01.677 INFO:teuthology.misc.health.smithi018.stderr:2: v1:172.21.15.18:6790/0 mon.c 2024-03-29T11:20:01.677 INFO:teuthology.misc.health.smithi018.stderr: 2024-03-29T11:20:01.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.693+0000 7f43714b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:01.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.693+0000 7f43714b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:01.677 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.693+0000 7f43714b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:20:01.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.693+0000 7f43714b3640 1 -- 172.21.15.18:0/2892437054 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f436c149940 con 0x7f436c0b1f60 2024-03-29T11:20:01.678 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.693+0000 7f4369ffb640 4 set_mon_vals no callback set 2024-03-29T11:20:01.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.694+0000 7f4363fff640 1 -- 172.21.15.18:0/2892437054 <== mon.2 v1:172.21.15.18:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2477700537 0 0) 0x7f435c001290 con 0x7f436c0b1f60 2024-03-29T11:20:01.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.695+0000 7f4363fff640 1 -- 172.21.15.18:0/2892437054 <== mon.2 v1:172.21.15.18:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3599602952 0 0) 0x7f435c084920 con 0x7f436c0b1f60 2024-03-29T11:20:01.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.695+0000 7f4363fff640 10 client.4691.objecter ms_dispatch 0x7f436c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:20:01.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.695+0000 7f4363fff640 3 client.4691.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:20:01.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.695+0000 7f4363fff640 3 client.4691.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:20:01.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.695+0000 7f43627fc640 10 monclient: get_auth_request con 0x7f434c06dd90 auth_method 0 2024-03-29T11:20:01.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.695+0000 7f4363fff640 20 client.4691.objecter dump_active .. 0 homeless 2024-03-29T11:20:01.679 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.695+0000 7f4363fff640 10 client.4691.objecter ms_handle_connect 0x7f434c06dd90 2024-03-29T11:20:01.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.703+0000 7f4363fff640 1 -- 172.21.15.18:0/2892437054 <== mon.2 v1:172.21.15.18:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f436c149940 con 0x7f436c0b1f60 2024-03-29T11:20:01.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.703+0000 7f4363fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:01.680 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:01.703+0000 7f4363fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:02.078 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:02.102+0000 7f43714b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:02.078 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:02.102+0000 7f43714b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:02.079 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:02.102+0000 7f43714b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.18:6790/0 2024-03-29T11:20:02.079 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:02.102+0000 7f43714b3640 1 -- 172.21.15.18:0/2892437054 --> v1:172.21.15.18:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f436c149940 con 0x7f436c0b1f60 2024-03-29T11:20:02.079 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:02.103+0000 7f4363fff640 1 -- 172.21.15.18:0/2892437054 <== mon.2 v1:172.21.15.18:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2084344819) 0x7f436c149940 con 0x7f436c0b1f60 2024-03-29T11:20:02.079 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:02.103+0000 7f4363fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:02.079 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:02.103+0000 7f4363fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:02.080 INFO:teuthology.misc.health.smithi018.stdout: 2024-03-29T11:20:02.080 INFO:teuthology.misc.health.smithi018.stdout:{"epoch":11,"fsid":"303809eb-4027-4a6a-aa53-d48fafbbf1ea","created":"2024-03-29T11:09:04.067083+0000","modified":"2024-03-29T11:19:08.996105+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:19:08.996105+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"02a13c46-dd11-4de5-8fbd-bb4aadf09403","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"4a8fc337-370b-4c2a-83d7-d5b6e9fb2436","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"9e0ff187-0cec-457f-9b8b-2c925ac96306","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"c74142fb-d4f2-4e57-a784-70d935727b13","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1d9bfdeb-5be9-4106-86dc-00841b3419f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"4d29e43e-9f99-450d-9d20-6b8dbf6eaa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d92168fd-c75c-43dd-b3dd-dcbf53d170d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"89acf17e-ff8f-40ad-bdcc-da3d92fb9738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:02.084 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:02.106+0000 7f43714b3640 10 librados: watch_flush enter 2024-03-29T11:20:02.084 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:02.106+0000 7f43714b3640 10 librados: watch_flush exit 2024-03-29T11:20:02.084 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:02.106+0000 7f43714b3640 20 client.4691.objecter shutdown clearing up homeless session... 2024-03-29T11:20:02.084 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:02.106+0000 7f43714b3640 10 client.4691.objecter successfully canceled tick 2024-03-29T11:20:02.084 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:02.107+0000 7f43714b3640 1 -- 172.21.15.18:0/2892437054 >> v1:172.21.15.70:6816/3101871562 conn(0x7f434c06dd90 legacy=0x7f434c070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:02.084 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:02.107+0000 7f43714b3640 10 monclient: shutdown 2024-03-29T11:20:02.084 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:02.107+0000 7f43714b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:02.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:02.107+0000 7f43714b3640 1 -- 172.21.15.18:0/2892437054 >> v1:172.21.15.18:6790/0 conn(0x7f436c0b1f60 legacy=0x7f436c13f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:02.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:02.107+0000 7f43714b3640 1 -- 172.21.15.18:0/2892437054 shutdown_connections 2024-03-29T11:20:02.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:02.107+0000 7f43714b3640 1 -- 172.21.15.18:0/2892437054 >> 172.21.15.18:0/2892437054 conn(0x7f436c066a00 msgr2=0x7f436c0b7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:02.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:02.107+0000 7f43714b3640 1 -- 172.21.15.18:0/2892437054 shutdown_connections 2024-03-29T11:20:02.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:02.107+0000 7f43714b3640 1 -- 172.21.15.18:0/2892437054 wait complete. 2024-03-29T11:20:02.085 INFO:teuthology.misc.health.smithi018.stderr:2024-03-29T11:20:02.107+0000 7f43714b3640 1 librados: shutdown 2024-03-29T11:20:02.098 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:02.099 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:20:02.100 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T11:20:02.100 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T11:20:02.100 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:20:02.136 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T11:20:02.136 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T11:20:02.136 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:20:02.206 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T11:20:02.207 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T11:20:02.207 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:20:02.272 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T11:20:02.273 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T11:20:02.273 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:20:02.343 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T11:20:02.344 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T11:20:02.344 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:20:02.380 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T11:20:02.380 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T11:20:02.380 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:20:02.451 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T11:20:02.451 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T11:20:02.451 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:20:02.526 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T11:20:02.526 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T11:20:02.526 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:20:02.604 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T11:20:02.604 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T11:20:02.604 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T11:20:02.604 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:20:02.628 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T11:20:02.628 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T11:20:02.628 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:20:02.670 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T11:20:02.670 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T11:20:02.670 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T11:20:02.670 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:20:02.682 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T11:20:02.682 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T11:20:02.682 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:20:02.733 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T11:20:02.734 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T11:20:02.734 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:20:02.745 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T11:20:02.745 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T11:20:02.746 DEBUG:teuthology.orchestra.run.smithi018:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T11:20:03.003 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi018.front.sepia.ceph.com 2024-03-29T11:20:03.003 DEBUG:teuthology.orchestra.run.smithi018:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T11:20:06.147 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi018.front.sepia.ceph.com 2024-03-29T11:20:06.147 DEBUG:teuthology.orchestra.run.smithi018:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T11:20:06.251 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi018.front.sepia.ceph.com 2024-03-29T11:20:06.251 DEBUG:teuthology.orchestra.run.smithi018:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T11:20:06.314 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi018.front.sepia.ceph.com 2024-03-29T11:20:06.314 DEBUG:teuthology.orchestra.run.smithi018:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T11:20:06.392 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi070.front.sepia.ceph.com 2024-03-29T11:20:06.392 DEBUG:teuthology.orchestra.run.smithi070:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T11:20:09.660 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi070.front.sepia.ceph.com 2024-03-29T11:20:09.660 DEBUG:teuthology.orchestra.run.smithi070:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T11:20:09.779 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi070.front.sepia.ceph.com 2024-03-29T11:20:09.780 DEBUG:teuthology.orchestra.run.smithi070:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T11:20:09.854 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi070.front.sepia.ceph.com 2024-03-29T11:20:09.855 DEBUG:teuthology.orchestra.run.smithi070:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T11:20:09.914 INFO:tasks.ceph:Archiving mon data... 2024-03-29T11:20:09.916 DEBUG:teuthology.misc:Transferring archived files from smithi018:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622677/data/mon.a.tgz 2024-03-29T11:20:09.916 DEBUG:teuthology.orchestra.run.smithi018:> mktemp 2024-03-29T11:20:09.934 INFO:teuthology.orchestra.run.smithi018.stdout:/tmp/tmp.E5qyFjP6U9 2024-03-29T11:20:09.935 DEBUG:teuthology.orchestra.run.smithi018:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.E5qyFjP6U9 2024-03-29T11:20:10.053 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod 0666 /tmp/tmp.E5qyFjP6U9 2024-03-29T11:20:10.140 DEBUG:teuthology.orchestra.remote:smithi018:/tmp/tmp.E5qyFjP6U9 is 212KB 2024-03-29T11:20:10.206 DEBUG:teuthology.orchestra.run.smithi018:> rm -fr /tmp/tmp.E5qyFjP6U9 2024-03-29T11:20:10.221 DEBUG:teuthology.misc:Transferring archived files from smithi018:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622677/data/mon.c.tgz 2024-03-29T11:20:10.221 DEBUG:teuthology.orchestra.run.smithi018:> mktemp 2024-03-29T11:20:10.275 INFO:teuthology.orchestra.run.smithi018.stdout:/tmp/tmp.xxNaGziPpD 2024-03-29T11:20:10.275 DEBUG:teuthology.orchestra.run.smithi018:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.xxNaGziPpD 2024-03-29T11:20:10.377 DEBUG:teuthology.orchestra.run.smithi018:> sudo chmod 0666 /tmp/tmp.xxNaGziPpD 2024-03-29T11:20:10.462 DEBUG:teuthology.orchestra.remote:smithi018:/tmp/tmp.xxNaGziPpD is 212KB 2024-03-29T11:20:10.555 DEBUG:teuthology.orchestra.run.smithi018:> rm -fr /tmp/tmp.xxNaGziPpD 2024-03-29T11:20:10.569 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622677/data/mon.b.tgz 2024-03-29T11:20:10.569 DEBUG:teuthology.orchestra.run.smithi070:> mktemp 2024-03-29T11:20:10.587 INFO:teuthology.orchestra.run.smithi070.stdout:/tmp/tmp.ixG1d943dL 2024-03-29T11:20:10.588 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.ixG1d943dL 2024-03-29T11:20:10.706 DEBUG:teuthology.orchestra.run.smithi070:> sudo chmod 0666 /tmp/tmp.ixG1d943dL 2024-03-29T11:20:10.794 DEBUG:teuthology.orchestra.remote:smithi070:/tmp/tmp.ixG1d943dL is 212KB 2024-03-29T11:20:10.887 DEBUG:teuthology.orchestra.run.smithi070:> rm -fr /tmp/tmp.ixG1d943dL 2024-03-29T11:20:10.903 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T11:20:10.903 DEBUG:teuthology.orchestra.run.smithi018:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:20:10.906 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:20:10.987 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:20:11.026 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:20:11.026 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T11:20:11.028 DEBUG:teuthology.misc:Transferring archived files from smithi018:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622677/remote/smithi018/crash 2024-03-29T11:20:11.029 DEBUG:teuthology.orchestra.run.smithi018:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:20:11.057 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622677/remote/smithi070/crash 2024-03-29T11:20:11.058 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:20:11.093 INFO:tasks.ceph:Compressing logs... 2024-03-29T11:20:11.094 DEBUG:teuthology.orchestra.run.smithi018:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:20:11.101 DEBUG:teuthology.orchestra.run.smithi070:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:20:11.176 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41452.log 2024-03-29T11:20:11.176 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38583.log 2024-03-29T11:20:11.177 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40694.log 2024-03-29T11:20:11.177 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.38583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40850.log 2024-03-29T11:20:11.177 INFO:teuthology.orchestra.run.smithi018.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41452.log.gz 2024-03-29T11:20:11.177 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40694.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T11:20:11.177 INFO:teuthology.orchestra.run.smithi018.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38583.log.gz 2024-03-29T11:20:11.177 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40343.log 2024-03-29T11:20:11.177 INFO:teuthology.orchestra.run.smithi018.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40694.log.gz 2024-03-29T11:20:11.177 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40811.log 2024-03-29T11:20:11.177 INFO:teuthology.orchestra.run.smithi018.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.40850.log.gz 2024-03-29T11:20:11.177 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41803.log 2024-03-29T11:20:11.177 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41959.log 2024-03-29T11:20:11.177 INFO:teuthology.orchestra.run.smithi018.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40343.log.gz 2024-03-29T11:20:11.178 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41803.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.40811.log.gz 2024-03-29T11:20:11.178 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39094.log 2024-03-29T11:20:11.178 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41959.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41803.log.gz 2024-03-29T11:20:11.178 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41686.log 2024-03-29T11:20:11.178 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38780.log 2024-03-29T11:20:11.178 INFO:teuthology.orchestra.run.smithi018.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41959.log.gz 2024-03-29T11:20:11.178 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41686.log: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.39094.log.gz 2024-03-29T11:20:11.178 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39601.log 2024-03-29T11:20:11.178 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.38780.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40889.log 2024-03-29T11:20:11.178 INFO:teuthology.orchestra.run.smithi018.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41686.log.gz 2024-03-29T11:20:11.179 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39601.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38780.log.gz 2024-03-29T11:20:11.179 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41062.log 2024-03-29T11:20:11.179 INFO:teuthology.orchestra.run.smithi018.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39601.log.gz 2024-03-29T11:20:11.179 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39484.log 2024-03-29T11:20:11.179 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40889.log: /var/log/ceph/ceph-client.admin.41062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34893.log 2024-03-29T11:20:11.179 INFO:teuthology.orchestra.run.smithi018.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40889.log.gz 2024-03-29T11:20:11.179 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39718.log 2024-03-29T11:20:11.179 INFO:teuthology.orchestra.run.smithi018.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.41062.log.gz 2024-03-29T11:20:11.179 INFO:teuthology.orchestra.run.smithi018.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39484.log.gz 2024-03-29T11:20:11.179 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.34893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39757.log 2024-03-29T11:20:11.179 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39211.log 2024-03-29T11:20:11.179 INFO:teuthology.orchestra.run.smithi018.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.34893.log.gz 2024-03-29T11:20:11.179 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39757.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40538.log 2024-03-29T11:20:11.179 INFO:teuthology.orchestra.run.smithi018.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39718.log.gz 2024-03-29T11:20:11.180 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39211.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39757.log.gz 2024-03-29T11:20:11.180 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39016.log 2024-03-29T11:20:11.180 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40538.log: gzip 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39211.log.gz -5 2024-03-29T11:20:11.180 INFO:teuthology.orchestra.run.smithi018.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40421.log 2024-03-29T11:20:11.180 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39562.log 2024-03-29T11:20:11.180 INFO:teuthology.orchestra.run.smithi018.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40538.log.gz 2024-03-29T11:20:11.180 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40421.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39016.log.gz 2024-03-29T11:20:11.180 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38739.log 2024-03-29T11:20:11.180 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39562.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41764.log 86.0% 2024-03-29T11:20:11.180 INFO:teuthology.orchestra.run.smithi018.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40421.log.gz 2024-03-29T11:20:11.180 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.38739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40499.log 2024-03-29T11:20:11.180 INFO:teuthology.orchestra.run.smithi018.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39562.log.gz 2024-03-29T11:20:11.180 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41764.log: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.38739.log.gz 2024-03-29T11:20:11.180 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39367.log 2024-03-29T11:20:11.181 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39952.log 2024-03-29T11:20:11.181 INFO:teuthology.orchestra.run.smithi018.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41764.log.gz 2024-03-29T11:20:11.181 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41569.log 2024-03-29T11:20:11.181 INFO:teuthology.orchestra.run.smithi018.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40499.log.gz 2024-03-29T11:20:11.181 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39952.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39367.log.gz 2024-03-29T11:20:11.181 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40616.log 2024-03-29T11:20:11.181 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38661.log 2024-03-29T11:20:11.181 INFO:teuthology.orchestra.run.smithi018.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39952.log.gz 2024-03-29T11:20:11.181 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41647.log 2024-03-29T11:20:11.181 INFO:teuthology.orchestra.run.smithi018.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41569.log.gz 2024-03-29T11:20:11.181 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.38661.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40616.log.gz 2024-03-29T11:20:11.181 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:20:11.181 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38858.log 2024-03-29T11:20:11.182 INFO:teuthology.orchestra.run.smithi018.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38661.log.gz 2024-03-29T11:20:11.182 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- 85.4% /var/log/ceph/ceph-client.admin.40460.log -- replaced with /var/log/ceph/ceph-client.admin.41647.log.gz 2024-03-29T11:20:11.182 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-29T11:20:11.182 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.38858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38622.log 2024-03-29T11:20:11.182 INFO:teuthology.orchestra.run.smithi018.stderr: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:20:11.182 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40460.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38858.log.gz 2024-03-29T11:20:11.182 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40070.log 2024-03-29T11:20:11.182 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.38622.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40460.log.gz 2024-03-29T11:20:11.182 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41296.log 2024-03-29T11:20:11.183 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40070.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.38622.log.gzgzip 2024-03-29T11:20:11.183 INFO:teuthology.orchestra.run.smithi018.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41491.log 2024-03-29T11:20:11.183 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41920.log 2024-03-29T11:20:11.183 INFO:teuthology.orchestra.run.smithi018.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40070.log.gz 2024-03-29T11:20:11.183 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41491.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41296.log.gz 2024-03-29T11:20:11.183 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41140.log 2024-03-29T11:20:11.183 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41920.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41491.log.gz 2024-03-29T11:20:11.183 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39250.log 2024-03-29T11:20:11.183 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41140.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T11:20:11.183 INFO:teuthology.orchestra.run.smithi018.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41920.log.gz 2024-03-29T11:20:11.183 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38505.log 2024-03-29T11:20:11.183 INFO:teuthology.orchestra.run.smithi018.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41140.log.gz 2024-03-29T11:20:11.183 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-osd.1.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39250.log.gzgzip 2024-03-29T11:20:11.184 INFO:teuthology.orchestra.run.smithi018.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38897.log 2024-03-29T11:20:11.184 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.38505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41179.log 2024-03-29T11:20:11.184 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.38897.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38505.log.gzgzip 2024-03-29T11:20:11.184 INFO:teuthology.orchestra.run.smithi018.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41218.log 2024-03-29T11:20:11.184 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41021.log 2024-03-29T11:20:11.184 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41179.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38897.log.gz 2024-03-29T11:20:11.184 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39328.log 2024-03-29T11:20:11.184 INFO:teuthology.orchestra.run.smithi018.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41179.log.gz 2024-03-29T11:20:11.184 INFO:teuthology.orchestra.run.smithi018.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41218.log.gz 2024-03-29T11:20:11.184 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38819.log 2024-03-29T11:20:11.184 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39835.log 2024-03-29T11:20:11.184 INFO:teuthology.orchestra.run.smithi018.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41021.log.gz 2024-03-29T11:20:11.185 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.38819.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39328.log.gz 2024-03-29T11:20:11.185 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38700.log 2024-03-29T11:20:11.185 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40148.log 2024-03-29T11:20:11.185 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39835.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38819.log.gz 2024-03-29T11:20:11.185 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.38700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39991.log 2024-03-29T11:20:11.185 INFO:teuthology.orchestra.run.smithi018.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.39835.log.gz 2024-03-29T11:20:11.185 INFO:teuthology.orchestra.run.smithi018.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38700.log.gz 2024-03-29T11:20:11.185 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41413.log 2024-03-29T11:20:11.185 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39991.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T11:20:11.185 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41413.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40148.log.gz 2024-03-29T11:20:11.185 INFO:teuthology.orchestra.run.smithi018.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39991.log.gz 2024-03-29T11:20:11.185 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40265.log 2024-03-29T11:20:11.186 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40031.log 2024-03-29T11:20:11.186 INFO:teuthology.orchestra.run.smithi018.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41413.log.gz 2024-03-29T11:20:11.186 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39640.log 2024-03-29T11:20:11.186 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40031.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40265.log.gz 2024-03-29T11:20:11.186 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40226.log 2024-03-29T11:20:11.186 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38544.log 2024-03-29T11:20:11.186 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39640.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40031.log.gz/var/log/ceph/ceph-client.admin.40226.log: 2024-03-29T11:20:11.186 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39874.log 2024-03-29T11:20:11.187 INFO:teuthology.orchestra.run.smithi018.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39640.log.gz 2024-03-29T11:20:11.187 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.38544.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40226.log.gz 2024-03-29T11:20:11.187 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40733.log 2024-03-29T11:20:11.187 INFO:teuthology.orchestra.run.smithi018.stderr:gzip/var/log/ceph/ceph-client.admin.39874.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.38936.log 2024-03-29T11:20:11.187 INFO:teuthology.orchestra.run.smithi018.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38544.log.gz 2024-03-29T11:20:11.187 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40733.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29073.log 2024-03-29T11:20:11.187 INFO:teuthology.orchestra.run.smithi018.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39874.log.gz 2024-03-29T11:20:11.187 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.38936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41608.log 2024-03-29T11:20:11.187 INFO:teuthology.orchestra.run.smithi018.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40733.log.gz 2024-03-29T11:20:11.187 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph.tmp-client.admin.29073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39055.log 2024-03-29T11:20:11.187 INFO:teuthology.orchestra.run.smithi018.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29073.log.gz 2024-03-29T11:20:11.187 INFO:teuthology.orchestra.run.smithi018.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38936.log.gz 2024-03-29T11:20:11.188 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39679.log 2024-03-29T11:20:11.188 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39055.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T11:20:11.188 INFO:teuthology.orchestra.run.smithi018.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41608.log.gz 2024-03-29T11:20:11.188 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39679.log: gzip -5 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39055.log.gz 2024-03-29T11:20:11.188 INFO:teuthology.orchestra.run.smithi018.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41842.log 2024-03-29T11:20:11.188 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35287.log 2024-03-29T11:20:11.188 INFO:teuthology.orchestra.run.smithi018.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39679.log.gz 2024-03-29T11:20:11.188 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41101.log 2024-03-29T11:20:11.188 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.35287.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41842.log.gz 2024-03-29T11:20:11.188 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39133.log 2024-03-29T11:20:11.188 INFO:teuthology.orchestra.run.smithi018.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35287.log.gz 2024-03-29T11:20:11.188 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39796.log 2024-03-29T11:20:11.188 INFO:teuthology.orchestra.run.smithi018.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.41101.log.gz 2024-03-29T11:20:11.189 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41257.log 2024-03-29T11:20:11.189 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39796.log: gzip -5 --verbose -- 86.6% /var/log/ceph/ceph-client.admin.41335.log 2024-03-29T11:20:11.189 INFO:teuthology.orchestra.run.smithi018.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39133.log.gz 2024-03-29T11:20:11.189 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41257.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39796.log.gz 2024-03-29T11:20:11.189 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40187.log 2024-03-29T11:20:11.189 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41257.log.gz 2024-03-29T11:20:11.189 INFO:teuthology.orchestra.run.smithi018.stderr: --/var/log/ceph/ceph-client.admin.41335.log: /var/log/ceph/ceph-client.admin.40772.log 2024-03-29T11:20:11.189 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40187.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41335.log.gz 2024-03-29T11:20:11.189 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-29T11:20:11.189 INFO:teuthology.orchestra.run.smithi018.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40187.log.gz 2024-03-29T11:20:11.189 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39445.log 2024-03-29T11:20:11.189 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose 86.2% -- /var/log/ceph/ceph-client.admin.41530.log 2024-03-29T11:20:11.189 INFO:teuthology.orchestra.run.smithi018.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40772.log.gz 2024-03-29T11:20:11.189 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39445.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41881.log 2024-03-29T11:20:11.190 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35184.log 2024-03-29T11:20:11.190 INFO:teuthology.orchestra.run.smithi018.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39445.log.gz 2024-03-29T11:20:11.190 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41530.log: /var/log/ceph/ceph-client.admin.41881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40982.log 2024-03-29T11:20:11.190 INFO:teuthology.orchestra.run.smithi018.stderr: 86.6%/var/log/ceph/ceph-client.admin.35184.log: -- replaced with /var/log/ceph/ceph-client.admin.41530.log.gz 2024-03-29T11:20:11.190 INFO:teuthology.orchestra.run.smithi018.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41881.log.gz 2024-03-29T11:20:11.190 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:20:11.190 INFO:teuthology.orchestra.run.smithi018.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35184.log.gz 2024-03-29T11:20:11.190 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40577.log 2024-03-29T11:20:11.190 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40982.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39172.log 2024-03-29T11:20:11.190 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40577.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40982.log.gz 2024-03-29T11:20:11.190 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T11:20:11.190 INFO:teuthology.orchestra.run.smithi018.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40577.log.gz 2024-03-29T11:20:11.191 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39172.log: gzip 92.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:20:11.191 INFO:teuthology.orchestra.run.smithi018.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40928.log 2024-03-29T11:20:11.191 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41725.log 2024-03-29T11:20:11.191 INFO:teuthology.orchestra.run.smithi018.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39172.log.gz 2024-03-29T11:20:11.191 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39913.log 2024-03-29T11:20:11.191 INFO:teuthology.orchestra.run.smithi018.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40928.log.gz 2024-03-29T11:20:11.191 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.41725.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41725.log.gz 2024-03-29T11:20:11.191 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39523.log 2024-03-29T11:20:11.191 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40304.log 2024-03-29T11:20:11.191 INFO:teuthology.orchestra.run.smithi018.stderr: 93.8%/var/log/ceph/ceph-client.admin.39913.log: -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T11:20:11.191 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39523.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39913.log.gz 2024-03-29T11:20:11.191 INFO:teuthology.orchestra.run.smithi018.stderr:gzip 86.9% -5 -- replaced with /var/log/ceph/ceph-client.admin.39523.log.gz 2024-03-29T11:20:11.191 INFO:teuthology.orchestra.run.smithi018.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40382.log 2024-03-29T11:20:11.192 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39289.log 2024-03-29T11:20:11.192 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40304.log: /var/log/ceph/ceph-client.admin.40382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39406.log 2024-03-29T11:20:11.192 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39289.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40304.log.gz 2024-03-29T11:20:11.192 INFO:teuthology.orchestra.run.smithi018.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40382.log.gz 2024-03-29T11:20:11.192 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40109.log 2024-03-29T11:20:11.192 INFO:teuthology.orchestra.run.smithi018.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39289.log.gz 2024-03-29T11:20:11.192 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40655.log 2024-03-29T11:20:11.192 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.39406.log: /var/log/ceph/ceph-client.admin.40109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38977.log 2024-03-29T11:20:11.192 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.40655.log: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.39406.log.gz 2024-03-29T11:20:11.192 INFO:teuthology.orchestra.run.smithi018.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.40109.log.gz 2024-03-29T11:20:11.192 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41374.log 2024-03-29T11:20:11.192 INFO:teuthology.orchestra.run.smithi018.stderr: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.40655.log.gz 2024-03-29T11:20:11.192 INFO:teuthology.orchestra.run.smithi018.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T11:20:11.192 INFO:teuthology.orchestra.run.smithi018.stderr:/var/log/ceph/ceph-client.admin.38977.log: /var/log/ceph/ceph-client.admin.41374.log: /var/log/ceph/ceph-mon.c.log: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.38977.log.gz 2024-03-29T11:20:11.193 INFO:teuthology.orchestra.run.smithi018.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41374.log.gz 2024-03-29T11:20:11.193 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T11:20:11.193 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34281.log 2024-03-29T11:20:11.193 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34401.log 2024-03-29T11:20:11.193 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.34281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34481.log 2024-03-29T11:20:11.193 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.34401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34441.log 2024-03-29T11:20:11.193 INFO:teuthology.orchestra.run.smithi070.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34281.log.gz 2024-03-29T11:20:11.193 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34241.log 2024-03-29T11:20:11.193 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.34481.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34401.log.gz 2024-03-29T11:20:11.194 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.34441.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:20:11.194 INFO:teuthology.orchestra.run.smithi070.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34481.log.gz 2024-03-29T11:20:11.194 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.34241.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T11:20:11.194 INFO:teuthology.orchestra.run.smithi070.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34441.log.gz 2024-03-29T11:20:11.194 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph.audit.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34241.log.gz 2024-03-29T11:20:11.194 INFO:teuthology.orchestra.run.smithi070.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T11:20:11.194 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T11:20:11.194 INFO:teuthology.orchestra.run.smithi070.stderr: 88.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:20:11.194 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34321.log 2024-03-29T11:20:11.194 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T11:20:11.195 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.34321.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:20:11.195 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T11:20:11.195 INFO:teuthology.orchestra.run.smithi070.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34321.log.gz 2024-03-29T11:20:11.195 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34201.log 2024-03-29T11:20:11.195 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34361.log 2024-03-29T11:20:11.195 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.34201.log: 87.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:20:11.195 INFO:teuthology.orchestra.run.smithi070.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34201.log.gz 2024-03-29T11:20:11.195 INFO:teuthology.orchestra.run.smithi070.stderr:/var/log/ceph/ceph-client.admin.34361.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34361.log.gz 2024-03-29T11:20:11.238 INFO:teuthology.orchestra.run.smithi070.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T11:20:22.803 INFO:teuthology.orchestra.run.smithi070.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T11:20:23.255 INFO:teuthology.orchestra.run.smithi070.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T11:20:23.279 INFO:teuthology.orchestra.run.smithi070.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T11:20:23.433 INFO:teuthology.orchestra.run.smithi018.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T11:20:27.401 INFO:teuthology.orchestra.run.smithi070.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T11:20:27.754 INFO:teuthology.orchestra.run.smithi018.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T11:20:29.173 INFO:teuthology.orchestra.run.smithi018.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T11:20:29.290 INFO:teuthology.orchestra.run.smithi018.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T11:20:59.445 INFO:teuthology.orchestra.run.smithi018.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T11:21:08.760 INFO:teuthology.orchestra.run.smithi018.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T11:21:08.762 INFO:teuthology.orchestra.run.smithi018.stderr: 2024-03-29T11:21:08.762 INFO:teuthology.orchestra.run.smithi018.stderr:real 0m57.649s 2024-03-29T11:21:08.762 INFO:teuthology.orchestra.run.smithi018.stderr:user 2m45.416s 2024-03-29T11:21:08.762 INFO:teuthology.orchestra.run.smithi018.stderr:sys 0m5.357s 2024-03-29T11:21:21.703 INFO:teuthology.orchestra.run.smithi070.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T11:21:21.705 INFO:teuthology.orchestra.run.smithi070.stderr: 2024-03-29T11:21:21.705 INFO:teuthology.orchestra.run.smithi070.stderr:real 1m10.555s 2024-03-29T11:21:21.705 INFO:teuthology.orchestra.run.smithi070.stderr:user 1m58.146s 2024-03-29T11:21:21.705 INFO:teuthology.orchestra.run.smithi070.stderr:sys 0m4.310s 2024-03-29T11:21:21.706 INFO:tasks.ceph:Archiving logs... 2024-03-29T11:21:21.706 DEBUG:teuthology.misc:Transferring archived files from smithi018:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622677/remote/smithi018/log 2024-03-29T11:21:21.707 DEBUG:teuthology.orchestra.run.smithi018:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:21:44.655 DEBUG:teuthology.misc:Transferring archived files from smithi070:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622677/remote/smithi070/log 2024-03-29T11:21:44.656 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:22:02.827 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:22:03.052 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=bd4817dba1e94400813b115975e60d39 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:22:03.056 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T11:22:03.071 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T11:22:03.080 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T11:22:03.080 DEBUG:teuthology.orchestra.run.smithi018:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:22:03.094 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:22:03.174 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T11:22:03.203 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T11:22:03.204 DEBUG:teuthology.orchestra.run.smithi018:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:22:03.207 DEBUG:teuthology.orchestra.run.smithi070:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:22:03.221 INFO:teuthology.orchestra.run.smithi018.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:22:03.223 INFO:teuthology.orchestra.run.smithi070.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:22:03.225 INFO:teuthology.orchestra.run.smithi018.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:22:03.225 INFO:teuthology.orchestra.run.smithi018.stdout:=============================================================================== 2024-03-29T11:22:03.225 INFO:teuthology.orchestra.run.smithi018.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 40 +436us[ +436us] +/- 82ms 2024-03-29T11:22:03.226 INFO:teuthology.orchestra.run.smithi018.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 40 -146us[ -146us] +/- 77ms 2024-03-29T11:22:03.226 INFO:teuthology.orchestra.run.smithi018.stdout:^* hv03.front.sepia.ceph.com 3 6 377 41 -239us[ -240us] +/- 76ms 2024-03-29T11:22:03.226 INFO:teuthology.orchestra.run.smithi018.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:22:03.227 INFO:teuthology.orchestra.run.smithi070.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:22:03.228 INFO:teuthology.orchestra.run.smithi070.stdout:=============================================================================== 2024-03-29T11:22:03.228 INFO:teuthology.orchestra.run.smithi070.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 38 +423us[ +423us] +/- 82ms 2024-03-29T11:22:03.228 INFO:teuthology.orchestra.run.smithi070.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 43 -156us[ -159us] +/- 77ms 2024-03-29T11:22:03.228 INFO:teuthology.orchestra.run.smithi070.stdout:^* hv03.front.sepia.ceph.com 3 6 377 41 -246us[ -250us] +/- 76ms 2024-03-29T11:22:03.228 INFO:teuthology.orchestra.run.smithi070.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:22:03.229 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T11:22:03.242 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T11:22:03.243 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T11:22:03.255 DEBUG:teuthology.orchestra.run.smithi018:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:22:03.339 DEBUG:teuthology.orchestra.run.smithi070:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:22:03.410 DEBUG:teuthology.orchestra.run.smithi018:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:22:03.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:22:03.442 DEBUG:teuthology.orchestra.run.smithi070:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:22:03.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:22:03.473 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T11:22:03.483 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T11:22:03.493 INFO:teuthology.task.internal:Duration was 1208.993711 seconds 2024-03-29T11:22:03.493 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T11:22:03.505 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T11:22:03.505 DEBUG:teuthology.orchestra.run.smithi018:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:22:03.508 DEBUG:teuthology.orchestra.run.smithi070:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:22:03.552 INFO:teuthology.orchestra.run.smithi018.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:22:03.558 INFO:teuthology.orchestra.run.smithi070.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:22:03.875 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T11:22:03.875 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi018.front.sepia.ceph.com 2024-03-29T11:22:03.875 DEBUG:teuthology.orchestra.run.smithi018:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:22:03.898 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi070.front.sepia.ceph.com 2024-03-29T11:22:03.899 DEBUG:teuthology.orchestra.run.smithi070:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:22:03.919 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T11:22:03.920 DEBUG:teuthology.orchestra.run.smithi018:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:22:03.942 DEBUG:teuthology.orchestra.run.smithi070:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:22:03.988 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T11:22:03.988 DEBUG:teuthology.orchestra.run.smithi018:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:22:04.013 DEBUG:teuthology.orchestra.run.smithi070:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:22:04.127 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T11:22:04.137 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T11:22:04.137 DEBUG:teuthology.orchestra.run.smithi018:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:22:04.164 DEBUG:teuthology.orchestra.run.smithi070:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:22:04.191 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T11:22:04.205 DEBUG:teuthology.orchestra.run.smithi018:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:22:04.207 DEBUG:teuthology.orchestra.run.smithi070:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:22:04.230 INFO:teuthology.orchestra.run.smithi018.stdout:kernel.core_pattern = core 2024-03-29T11:22:04.257 INFO:teuthology.orchestra.run.smithi070.stdout:kernel.core_pattern = core 2024-03-29T11:22:04.281 DEBUG:teuthology.orchestra.run.smithi018:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:22:04.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:22:04.308 DEBUG:teuthology.orchestra.run.smithi070:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:22:04.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:22:04.335 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T11:22:04.345 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T11:22:04.345 DEBUG:teuthology.misc:Transferring archived files from smithi018:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622677/remote/smithi018 2024-03-29T11:22:04.346 DEBUG:teuthology.orchestra.run.smithi018:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:22:04.385 DEBUG:teuthology.misc:Transferring archived files from smithi070:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622677/remote/smithi070 2024-03-29T11:22:04.385 DEBUG:teuthology.orchestra.run.smithi070:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:22:04.422 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T11:22:04.422 DEBUG:teuthology.orchestra.run.smithi018:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:22:04.425 DEBUG:teuthology.orchestra.run.smithi070:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:22:04.471 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T11:22:04.484 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T11:22:04.484 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T11:22:04.497 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T11:22:04.498 DEBUG:teuthology.orchestra.run.smithi018:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:22:04.500 DEBUG:teuthology.orchestra.run.smithi070:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:22:04.512 INFO:teuthology.orchestra.run.smithi018.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:22 /home/ubuntu/cephtest 2024-03-29T11:22:04.525 INFO:teuthology.orchestra.run.smithi070.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:22 /home/ubuntu/cephtest 2024-03-29T11:22:04.526 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T11:22:04.539 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T11:22:04.630 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T11:22:04.657 DEBUG:teuthology.nuke:shortname: smithi018 2024-03-29T11:22:04.658 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:22:04.682 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi018.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622677', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:53:50.297642', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIahkFEMlmnbD1gAQSJHalvyf71pZc2mxw+81x+ayQCdypgeL+kns4j8XKqBYveluo9sNsk0UJX9n5RTUJZ0Mnk='} 2024-03-29T11:22:04.685 DEBUG:teuthology.nuke:shortname: smithi070 2024-03-29T11:22:04.685 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:22:04.706 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi070.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622677', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:53:50.299627', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJrL8s29ynEsIThArj/rDyBt8CR2iFx2u+RvNeCjnlMFTVVtqNRzhhtNViVWYO5QIGCiXPV536ehJiwSbhIipA0='} 2024-03-29T11:22:04.736 INFO:teuthology.orchestra.console.smithi018:Power off 2024-03-29T11:22:04.736 DEBUG:teuthology.orchestra.console.smithi018:pexpect command: ipmitool -H smithi018.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:22:04.760 DEBUG:teuthology.orchestra.console.smithi018:power off output: Chassis Power Control: Down/Off 2024-03-29T11:22:04.760 DEBUG:teuthology.orchestra.console.smithi018:pexpect command: ipmitool -H smithi018.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:22:04.776 INFO:teuthology.orchestra.console.smithi070:Power off 2024-03-29T11:22:04.776 DEBUG:teuthology.orchestra.console.smithi070:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:22:04.788 DEBUG:teuthology.orchestra.console.smithi018:check power output: Chassis Power is on 2024-03-29T11:22:04.800 DEBUG:teuthology.orchestra.console.smithi070:power off output: Chassis Power Control: Down/Off 2024-03-29T11:22:04.800 DEBUG:teuthology.orchestra.console.smithi070:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:22:04.824 DEBUG:teuthology.orchestra.console.smithi070:check power output: Chassis Power is on 2024-03-29T11:22:08.789 DEBUG:teuthology.orchestra.console.smithi018:pexpect command: ipmitool -H smithi018.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:22:08.825 DEBUG:teuthology.orchestra.console.smithi070:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:22:08.900 DEBUG:teuthology.orchestra.console.smithi018:check power output: Chassis Power is on 2024-03-29T11:22:08.934 DEBUG:teuthology.orchestra.console.smithi070:check power output: Chassis Power is on 2024-03-29T11:22:12.902 DEBUG:teuthology.orchestra.console.smithi018:pexpect command: ipmitool -H smithi018.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:22:12.936 DEBUG:teuthology.orchestra.console.smithi070:pexpect command: ipmitool -H smithi070.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:22:13.012 DEBUG:teuthology.orchestra.console.smithi018:check power output: Chassis Power is off 2024-03-29T11:22:13.045 DEBUG:teuthology.orchestra.console.smithi070:check power output: Chassis Power is off 2024-03-29T11:22:13.113 INFO:teuthology.orchestra.console.smithi018:Power off completed 2024-03-29T11:22:13.146 INFO:teuthology.orchestra.console.smithi070:Power off completed 2024-03-29T11:22:13.252 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} duration: 1208.9937109947205 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=bd4817dba1e94400813b115975e60d39 status: fail success: false 2024-03-29T11:22:13.253 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:22:13.333 INFO:teuthology.run:FAIL